July 02, 2015

Last week, the Supreme Court gave us a fantastic reason to celebrate, legalizing gay marriage in the U.S. At Twilio HQ, we celebrated alongside our friends and co-workers, in spirit with hundreds of thousands of men and women across the country and abroad.

Twitter put out a super easy way to add the signature rainbow heart to your tweet with the hashtag #LoveWins. Facebook offered an easy to add a rainbow photo filter on your profile picture. Now, Abe Diaz created another way you can show your pride.

Using Twilio MMS, you can text in any photo to 415 360 2438 and Abe’s service PrideFy will send you that photo back with a rainbow filter added.

PrideFyPhoto

We’re happy (and humbled) to syndicate Abe’s post below. Check out the original post on Abe’s blog here.

Sharing Your Pride via Twilio MMS with PrideFy

Inspired by the decision of the Supreme Court of the United States to declare same sex marriage a constitutional right and the celebratepride tool that Facebook launched on that same day, I decided to create an MMS ( Twilio powered) version of the same for those who (and I’m not sure why) do not have a Facebook account yet. Here are the steps I followed to make this happen; if you want to see the demo go to Pridefy.com

Step 1: Change Opacity
First I needed to find a way to change the opacity of the rainbow flag; a 50% seemed like the right choice here but I made this option part of the input for the method, just in case you want to play with this in the future. Check this article for some more details into this.

public static Bitmap ChangeOpacity(Image img, float opacityvalue)
        {
            Bitmap bmp = new Bitmap(img.Width, img.Height);
            Graphics graphics = Graphics.FromImage(bmp);
            ColorMatrix colormatrix = new ColorMatrix();
            colormatrix.Matrix33 = opacityvalue;
            ImageAttributes imgAttribute = new ImageAttributes();
            imgAttribute.SetColorMatrix(colormatrix, ColorMatrixFlag.Default, ColorAdjustType.Bitmap);
            graphics.DrawImage(img, new Rectangle(0, 0, bmp.Width, bmp.Height), 0, 0, img.Width, img.Height, GraphicsUnit.Pixel, imgAttribute);
            graphics.Dispose();  
            return bmp;
        }

I tried this method and got some results, but not quite what I wanted. See what I mean here:
output1
Background Image with Overlay at 50%
 

Step 2: Scale Image

I wanted like the rainbow flag to cover the whole image, So I tried making both images the same size and this is what I got:
output2
Stretched with Uniform Ratios

This didn’t turn out like I wanted it. I needed to scale the image using ratios horizontally but needed to retain the same vertical height.

public static Image ScaleImage(Image image, int maxWidth, int maxHeight)
{
var ratioX = (double)maxWidth / image.Width;
var ratioY = (double)maxHeight / image.Height;
var ratio = Math.Min(ratioX, ratioY);
 
var newWidth = (int)(image.Width * ratio);
var newImage = new Bitmap(newWidth, maxHeight);//
Graphics.FromImage(newImage).DrawImage(image, 0, 0, newWidth, maxHeight);//
return newImage;
}

Check the results now:
output3

Step 3 – Putting it all together for Image Manipulation

When I receive an MMS message what I get is not the actual file (binary) but a link to the media file in Twilio’s servers. Therefore I needed to do a webrequest to get the media file (this is the MediaUrl0 parameter). I grabbed the flag image from a public Flickr url.

private static Image CreateBlendedImage(string backgroundimageurl)
{
Image imageBackground;
Image imageOverlay;
var request = WebRequest.Create(backgroundimageurl);
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
{
imageBackground = Bitmap.FromStream(stream);
}
var request2 = WebRequest.Create("http://c1.staticflickr.com/1/286/19041020628_2657a057cc_k.jpg"); //Rainbow Flag
using (var response2 = request2.GetResponse())
using (var stream2 = response2.GetResponseStream())
{
imageOverlay = Bitmap.FromStream(stream2);
}
 
imageOverlay = ChangeOpacity(imageOverlay, 0.5F);
imageOverlay = ScaleImage(imageOverlay, imageBackground.Width, imageBackground.Height);
imageOverlay = imageOverlay.GetThumbnailImage(imageBackground.Width, imageBackground.Height, null, IntPtr.Zero);
 
Image img = new Bitmap(imageBackground.Width, imageBackground.Height);
using (Graphics gr = Graphics.FromImage(img))
{
gr.DrawImage(imageBackground, new Point(0, 0));
gr.DrawImage(imageOverlay, new Point(0, 0));
}
return img;
}

I then do the scaling, overlay and return the (in memory) image to the main controller.

Step 4 – The MMS Controller

I created an MVC controller to handle the incoming images and return back TwiML markup (Note: This controller must inherit from TwilioController in order to be able to send that TwiML markup back). You can do the same with a WebAPI controller, Node, etc. I just went for simple here

The same way that Twilio sends images via MMS, I needed to send them back; this means I had to save the image somewhere and then send the URL to that media file back. To do this I used Azure Blob Storage; again nothing fancy here, check this tutorial for a quick how to. I created a container for my images,created a blob and uploaded the image to that blob.

//Reminder this class needs to inherit from TwilioController --> public class MMSController : TwilioController
public ActionResult Index(string from, string mediaurl0)
{
Image blended = CreateBlendedImage(mediaurl0);
//We are using Azure Storage, and loading our credentials from our config file
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings[""].ToString());
// Create the blob client
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.GetContainerReference("");
//Allow public access to the blob (but not to the container)
container.SetPermissions(new BlobContainerPermissions {PublicAccess = BlobContainerPublicAccessType.Blob });
// Create the container if it doesn't already exist.
container.CreateIfNotExists();
// Retrieve reference to a blob names using the from phone number.
//We add the png file extension so its easier for us when looking at a list of blobs
CloudBlockBlob blockBlob = container.GetBlockBlobReference(from + ".png");
// Create or overwrite the phone mumber blob with contents from a local file.
//We re using temp file space for file creation
var filepath = System.IO.Path.GetTempFileName();
blended.Save(filepath, ImageFormat.Png);
//remmeber to set the Content Type, else Azure will return appliction/octet and Twilio won't be able to read the image
blockBlob.Properties.ContentType = "image/png";
//Upload the file to Azure storage
blockBlob.UploadFromFile(filepath, FileMode.Open);
//Create th Twilio TwiML markup response
var response = new TwilioResponse();
string[] mediaurls = new string[1];
//For some reason only http worked for me
mediaurls[0] = blockBlob.Uri.AbsoluteUri.Replace("https://","http
//Add the Blended Image URL from Azure to the response
response.Message(mediaurls);
//Note how we return a TwiML respone and not a View
return TwiML(response);
}

Warning: don’t forget to set the Content-Type for the blob, failure to do this will serve the blob as “application/octet” which Twilio won’t be able to handle.

Hopefully you enjoyed this little project, I surely did. Feel free to reach out if you have any questions with this demo.

Share Your Pride With Twilio MMS and Pridefy.com

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


July 01, 2015

Today we’re releasing Twilio’s first transparency report detailing requests for customer information by municipal, state, provincial and federal governments globally. As part of our commitment to the privacy of your data, this report represents the start of a program to produce clear visibility to the Twilio community around these requests.

In this spirit, this report’s objective is to inform you of the total volume of government requests for information received by Twilio, how Twilio responded to the requests and how often Twilio notified users of the requests.

  • You can view a PDF of the entire report here
  • You can download a text file of the report from our GitHub repository where we will store all future reports here.
  • Read our statement on privacy here.

What This Report Means

Twilio powers communications ranging from voice to messaging to video. As a communications platform relied on by tens of thousands of companies and hundreds of thousands of developers, we receive requests from governmental agencies for customer or end user information in a fashion similar to many communications backbones or cloud platforms.

Who Requests Information

These information requests come from many agencies representing municipal, state, provincial, and federal governments worldwide and can include:

  • Law enforcement agencies
  • Courts
  • Regulatory bodies with subpoena authority

What Kind of Requests are Made

These information requests take the form of discovery instruments carrying the force of law in the regions where Twilio operates, such as:

  • Civil Investigative Demand
  • Court Order
  • Grand Jury Subpoena
  • Subpoena
  • Search Warrant
  • Police Force Order
  • Exigent, Emergency and/or Child Safety Circumstances
  • Record Preservation Request
  • Takedown Request
  • National Security Letters
  • Foreign Intelligence Surveillance Act (FISA) Court Orders

The Importance of Privacy, The Process of an Information Request

We know how important transparent data security is for any developer in their usage of cloud services. For this reason, we stringently uphold our privacy and transparency standards when complying with governmental information requests. This extends not only to how we reply to these requests, but how we share our reply process with our users.

We notify our customers that we have complied with a government request whenever we are not specifically prohibited from doing so by statute, warrant, subpoena or court or administrative order. This results in one of four outcomes:

  • Twilio declines to produce information in response
  • Agency withdraws request for information when informed of our customer disclosure policy
  • Twilio provides customer contact information only
  • Twilio provides customer contact information and specific user content only as indicated in the applicable request

In our first transparency report, you will find that we received 268 requests for information during the first half of 2015. The number of agencies, number of request types and disclosure outcomes are available in aggregate. View the full report here.

Going Forward

Trust is not built through a single act. It is the cumulative result of a series of actions and best practices. The trust you place in Twilio for your vital communications is gravely important to us – this transparency program is just another step in providing you the visibility you deserve and expect from a cloud service. Reports on governmental information requests will continue twice a year both on our website and in a public source code management repository for easy and transparent consumption of any modifications.

Thank you for reading this report and putting that trust in Twilio. If you have any questions about the security of your account or data, please reach out to our support team at help@twilio.com. This report will evolve over time with your feedback – please reach out to us at transparency@twilio.com with specific questions or feedback on this report.

Transparency Report for Government Requests for Customer Information

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


June 30, 2015

JavaScript is eating the programming world. Look around and you’ll see JavaScript in places people never would’ve expected 10 years ago. There’s JavaScript running on servers. There are more front-end JavaScript frameworks than you can shake a stick at. And there’s even JavaScript on my dog! Lucky me because I love JavaScript.

A few months ago at BrooklynJS I was super excited to discover a new way I can use JavaScript. Alan Mooiman gave a great talking showing how you can use JavaScript to automate your Mac. After seeing Alan’s talk I wondered, could I write a script to take some action on my Mac via SMS? Spoiler alert: yes! yes I can!

This post is all about learning how to automate your Mac via SMS message. The actual task we automate could be one of countless options but for the sake of simplicity we’re going to write an app that sends an iMessage message from the messages app when an SMS comes in.

Our Tools

  • Node.js – We’ll be using node to receive our incoming SMS message from Twilio.
  • JavaScript for Automation – It’s like AppleScript but it’s JavaScript.
  • OSX >= 10.10 – JavaScript for Automation requires Yosemite or higher.
  • Twilio – Need an account? Sign up for free!

Sending iMessages with JavaScript
We’ll start by working on the automation portion of our application. We’ll use JavaScript for Automation to make this happen. JavaScript for Automation allows you to do the things you would historically use AppleScript for using JavaScript. Pretty rad, right?

Writing this in Apple’s Script Editor will make it easy for us to quickly test our code. If you’re a Vim lover like me, I know it’ll be tough but you can do it!

Once we open Script Editor we need to switch our language from AppleScript to JavaScript.

Now we can start writing our code. Since we’re keeping it basic we only need 3 lines of code:

var messages = Application('Messages');

var buddy = messages.services["E:%REPLACE_WITH_YOUR_IMESSAGE_EMAIL%"].buddies["%REPLACE_WITH_BUDDYS_EMAIL%"];

messages.send("JavaScript sent this message!", {to: buddy});

Make sure your replace the first element with the e-mail address you use for iMessage but keep the preceding “E:”. For example, mine would be “E:ricky@twilio.com”. Are you unsure what your iMessage email is? In the Messages application press “cmd ,” and then click on the accounts tab. Replace the second element with the address of a buddy you’d like to message or if you don’t want to spam your friend during testing you can send a message to yourself.

Before we run this code let’s break it down. On the first line we’re getting an object that references the Messages application. The second line we get a reference to our buddy that we want to send a message to. On the last line, we call the send method and pass it the message content we want to send as well as the buddy we’re sending to.

Now that we have our code in place we can run it by hitting the big play button in Script Editor (or if you want to be l33 by pressing cmd r). You’ll see your message application open and a message is sent to the friend you specified. Oh yeah!

Hooking It Up To Twilio
Now that we’ve automated our mac to send our iMessage it’s time to write the code that triggers this automation when a text message comes in via Twilio.

First thing’s first, we need to create a new node application. Hop into Terminal and get this bad boy started:

mkdir sms-automation
cd sms-automation
npm init

We’re going to be using hapi.js for this application. If you haven’t worked with hapi before, it’s a framework that makes building node.js web applications super easy. If you’ve worked with express it is a lot like that! In order to use hapi we need to install it:

npm install hapi

Now we can create our index.js file that will contain the code for our basic application:

var Hapi = require('hapi');

var server = new Hapi.Server();
server.connection({
    host: 'localhost',
    port: 8000
});

server.start();

We’ve written the code for a basic server but right now it doesn’t do anything. Luckily, we just need one route for our application. We can call the route incoming:

server.route({
    method: ‘POST’,
    path: '/incoming’,
    handler: function (request, reply) {
       reply('hello world');
    }
});

server.start();

Start up our server and test out our new route:

node index.js

You may have noticed that we made our route only respond to POST requests. We can use curl to try this out:

curl -X POST http://127.0.0.1:8000/incoming

We now see we’re getting our beautiful “hello world” response. But we want to do something that receives information about our incoming message and then executes our JavaScript that sends the message. In order to use that we’re going to need to use the exec() function. Warning! The exec function runs a shell command on your local machine so be careful that you make sure users can’t do anything malicious if you open this up to the outside world.
In order to use exec() we need to require a couple standard libraries at the top of our index.js file:

var sys = require('sys'),
    exec = require('child_process').exec;

Before we write the next part we need to jump back into Script Editor and save the code we wrote earlier in the same directory as our Node.js application. Save it as “sendmessage.scpt”. Saving a file sounds like it should be easy but the Script Editor dialogue can be confusing. When you go to save you need to click on the down arrow next to the “Save As” name and you’ll have the ability to browse to your folder.

So far we’ve only run our automation script via Script Editor but we can also run it from the command line. Jump into terminal and browse the directory where we just saved our file. Now run this command:

osascript -l JavaScript ./sendmessage.scpt

Again, you’ll see our messages application has sent our iMessage for us. This command is fairly basic but I wanted to highlight -l argument. This argument indicates that the script we’re running is JavaScript, otherwise the command will think we’re running code written in AppleScript and it will crash.

We now have everything in place to call this script from our Node.js application. Go back into index.js and add the following code to call JavaScript for Automation script:

    handler: function (request, reply) {
        exec("osascript -l JavaScript "   __dirname   "/sendmessage.scpt", function(err, stdout, stderr) {
         console.log(err);
    });
       reply('hello world');
    }

This route will get called by Twilio whenever someone sends an SMS message to our app. We want to respond to that message indicating that our SMS was received. We can do that using TwiML. TwiML allows us to provide some basic instructions of what Twilio should do with an incoming message or phone call. We could use the Twilio Node.js helper library to generate our TwiML but since TwiML is actually XML we’re going to just hardcode in our response to avoid adding another dependency:

    path:'/incoming',
    handler: function (request, reply) {
       exec("osascript -l JavaScript "   __dirname   "/sendmessage.scpt", function(err, stdout, stderr) {
        console.log(err);
       });
       reply('<Response><Message>Thanks! Sending iMessage now!</Message></Response>');
    }

Twilio needs to be able to access our application so we need to expose our local host to the outside world. Fortunately, we can use one of my favorite tools called ngrok to make this happen. If you haven’t used this tool before, download and install ngrok.

With ngrok installed, we can run the following command to expose our localhost to the outside world:

ngrok http 8000

Make note of the url ngrok created for your application then head to your Twilio dashboard. If you don’t already have a phone number you’d like to use for your application then buy a new phone number. Once you have a phone number set the SMS Request URL to the /incoming route on our application:

ngrok
The moment of truth. Send an SMS message to your application and watch as your Messages app opens and sends an iMessage. Party time!

You can find the completed code for our application on GitHub.

Hackers Gonna Hack
You’ve now built a basic example showing how you can automate your Mac via SMS. And we’re doing it all using JavaScript! What will you build now that you have this ability? Here are a couple ideas:

  • SMS to have your computer open Photobooth snap a picture and send back to you via MMS
  • SMS your computer and get it to say something (using terminal say) to scare people in your house

Are you looking for a way to improve the iMessage application we built together today? Try replacing the hard coded text we’re sending each time with the body of the SMS message you’re sending in.

Have any questions or what to show me the latest project you’ve hacked together? Drop a note in the comments below or holler at me on twitter (@rickyrobinett).

 

Automate Your Mac Via SMS using JavaScript for Automation, Node.js and Twilio

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


June 29, 2015

“Hey Sam, I think we might actually break 1000 attendees this time.”

One of the new organizers told me as I gazed in awe at the last HackRU of my student career. This event and community have come so far in just a few short years. The first HackRU had more boxes of pizza than human beings in attendance so the thought of 1000 people showing up blew my mind. I was proud to be part of the team that could grow this community into what it is today.

My thoughts were interrupted when I encountered Rob Spectre walking around. During the previous week, I had accepted an offer to join Twilio’s devangelism team. Rob, the leader of this team, asked for my help assembling the Twilio banner. I looked around and realized that it was because of this community that I was led to developer evangelism. This community made me who I am now. From every community I’ve contributed to, I’ve gotten back even more than what I put into helping that community grow.

How did this happen?

Three Years Earlier

I was exhausted and had a pile of schoShredding at the Court Tavern in New Brunswick, NJolwork to get through for finals week. Instead, I decided to spend that Saturday with my band Condition Critical. We were prepping for a show with hardcore punk/thrash legends D.R.I. (Dirty Rotten Imbeciles). After hours of shredding through our set, I decided to check my phone.

“Are you going to that RU hackathon thing?”

It was a text from my buddy Reggie. The idea of HackRU seemed exciting to us and we really wanted to check it out so after seeing Reggie’s message, I left practice and scrambled to catch a bus back to New Brunswick. Upon arriving, I was amazed by the vibrant community of developers thriving right under my nose. I had no idea that this collection of builders and hackers existed at my school. It made me realize that I didn’t need to limit myself to only learning from my Computer Science courses and inspired me to start building things on my own. This hackathon changed my life.

Growing the Rutgers Hacker Community

Rutgers had a reputation for having top notch hackers and a thriving developer community. Rumors were spreading that this was ending as the next school year rolled around because the best hackers were graduating. Now the up-and-comers had to take things into their own hands, myself included. After going to 10 hackathons in one semester and taking a job tutoring CS students at our de facto hacker hangout (“The CAVE”), I understood the community better.

My involvement with our hacker community led me to helping with the next HackRU and ultimately to volunteering as the new director of the event. We were the final of five events for the first Major League Hacking season, which measured schools based on their attendance and performance at hackathons. We were overwhelmed with excitement upon hearing that Rutgers took second place out of 110 schools. My first semester leading HackRU was a success and our community grew immensely. But my community building skills grew even more through the experience I’ve gained working non-stop on the event.

New Home, New Passion, New York

During the next Winter break, I vividly remember walking along a beach in Cape May frantically trying to check my email despite not having service. When I finally got through, I began jumping for joy after seeing a message titled “Your hackNY Application – YES!”

hackny2014The hackNY fellowship pairs college hackers with Summer internships at NY based startups and provides a rad speaker series and awesome mentorship environment. Numerous fellows in the past have had some awesome exits, so the stakes were high. My leadership with HackRU and involvement in the collegiate hackathon scene helped me immensely when it came to my hackNY application. This program skyrocketed my technical growth and propelled me into the NYC tech community that I fell in love with. Many of the strongest friendships I have in New York were made through hackNY. I reach out to this community on a regular basis whenever I need technical or life advice.

Through hackNY, I worked at Ordr.in, a small startup with a food delivery API. I worked on some internal tools and open source projects related to the API, but my favorite aspect of being with Ordr.in was my ability to work on a product that serves developers. Seeing “the spark” in the eyes of fresh developers getting their first app working and inspiring them to continue hacking was the most rewarding feeling I’ve ever had.

The feeling of seeing that spark was what made me so excited when Ordr.in asked me to be their devangelist while finishing up my last year of school. Throughout my time at hackathons I have always admired the developer evangelists attending these events. They were superheroes to me. Errors that would take me hours to solve were fixable within minutes of having an extra set of eyes. The best ones inspired developers in the same way that the leaders in the Rutgers hacker community set out to do.

Back to the Future

I was running around helping to coordinate the massive amounts of Chinese food that had just arrived, when I was notified that our attendance count finally hit 1000. I was astounded to have helped HackRU reach an inflection point where it grew organically to its current state. I am lucky to be part of this community and to have grown so much because of it. The leadership experience I’ve gained from running HackRU and the mentorship I received from hackNY and Ordr.in put me in my current position. All of this is what compelled me to inspire and equip developers in my career.

It is amazing to see what happens when a developer harnesses the ability to code. As if possessing magical powers, crazy ideas can become reality in mere hours. I’m looking forward to supercharging the developer community in the greatest city in the world and beyond. Feel free to reach out if you ever need help with anything Twilio or code related. Or if you just want to jam and share your story.

Introducing Twilio Developer Evangelist Sam Agnew

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


Last November we announced the unbundling of our PSTN connectivity as a service.  Elastic SIP Trunking provides instant global voice provisioning for your IP infrastructure (e.g. IP-PBX, SBC, UC hardware, VoIP gateway, etc). Twilio uses the power of the cloud to offer resilience, intelligent local call routing, sms-enabled trunks, cloud-based call recording, multi-tenancy, and full WebRTC support.

Since November, we have seen companies using this connectivity to provide great multi-channel customer support experiences, create engaging advertising campaigns, change the way business’ collaborate, and deliver critical alerts. In the past 3 months, Elastic SIP Trunking traffic grew by nearly 400% thanks to companies augmenting their businesses and services with cloud based PSTN connectivity.

SIP Trunking Pricing Changes

Beginning July 1st we are making two changes to our Elastic SIP Trunking prices. We are introducing a new zoned pricing model and we are lowering our North American termination price by 12.5% from $0.008/min to $0.007/min.

Our primary price will be known as Zone 1 and will be priced at $0.007.  88% of all US and Canadian prefixes fall into Zone 1 and 95% of aggregate Twilio voice traffic falls in Zone 1. Elastic SIP Trunking customers will automatically save 12.5% on termination to US/CA – Zone 1. Our continued growth allows us to achieve greater economies of scale and continue our tradition of passing cost savings to customers in the form of lower prices.

We are restructuring our SIP Trunking pricing zones which today are US/CA, Alaska, Hawaii, and Other. We will be offering the same service, but now defining our pricing Zones as US/CA (Zone 1) and United States – Extended Zone 2, 3, & 4. These zones are now defined by prefix (NPA-NXX) and the Extended Zones will include remote and rural regions. Based on customer feedback, we are also providing a rate deck with all of our pricing on a per NPA-NXX basis.

We remain focused on bringing you the most compelling platform for communications, whether you’re building your entire service on the Twilio platform or connecting an existing service to the PSTN.

Announcing Lower SIP Trunking Prices & New Pricing Zones

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


June 26, 2015

In the morning you can find me scouring newswires and wading through press releases about the latest and greatest speech products. This technology represents a major breakthrough for the enterprise, that solution provides instantaneous ROI. Reporting about the commercial side of speech technology is my bread and butter, but what truly excites me is when […]

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


June 24, 2015

Guiding a patient from sickness to health takes a team. Hospital staff, pharmacy staff, friends,and family all work to help that person do what they need to do to get better. But, getting better is a process that requires managing appointments, highly time sensitive medication schedules and other timely logistics. If a patient gets behind on their schedule, it can cost them their health, lead them back to another expensive hospital visit, and leave them reordering pricy medication.

CareSpeak helps make the coordination between all these different parties, at all these different times seamless. There’s one medium they chose to do it – SMS.

CareSpeak works to connect hospitals, pharmacies and patients together to ensure proper treatment. Patients can miss a piece of mail the hospital sends them, reminding them of their medication schedule. But they’re far less likely to miss a text reminder via Twilio SMS. Patients are even less likely to get off their medication schedule when they know that there’s a CareSpeak trained pharmacist on the other end of the line who can answer their questions.

Patients have all sorts of questions about their medication that are easily answered via text through CareSpeak. If they want to know how to store medication, how to take it, when to take it, or what possible side effects might be – a Google search doesn’t cut it. The personal touch patients receive when using CareSpeak to connect with pharmacists is incredibly effective.

CareSpeak founder, Serge Loncar, believes in a personal and mobile approach to health care. Using Twilio, CareSpeak can offer more to not only patients, but the healthcare industry itself.

“Twilio is incredibly reliable,” said Loncar.“The platform caters to people who need to do something fast and on the fly and at scale. I’m a marketing person and I can provision a number.”

Not only can Loncar provision a number, he can provision it wherever he needs with Twilio’s global availability. Loncar travels all over the globe giving CareSpeak demos. In his live demos to potential customers, he’s always confident he can send them a text via Twilio, no matter where they are in the world.

Looking forward, Loncar plans to expand aggressively into new international markets. CareSpeak CTO, Krešimir Nesek relies on the Twilio backbone to make sure their vision becomes a reality. ““All the API tools can help us plan for the future. I love the simplicity and flexibility of Twilio,” says Nesek.

CareSpeak Makes Outpatient Care Seamless For Patients and Providers via SMS

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


With budgets and costs constantly at the forefront of business decisions, I thought it would be interesting to target an area where costs are thought to be optimized, where, ostensibly, there are few hidden costs, and where even some of the best CFOs overlook value to the company.

IVRs Are Simply Necessary
Call centers and IVRs are often seen as necessary: a necessary evil by customers, a necessary cost by companies, and a place where customer service challenges necessarily abound. Call centers often come with an expensive price tag both to launch and to maintain. Overhead costs can be through the roof (pun intended), especially if there are high volumes of complex calls because they require the attention of human agents. To help offset the cost of expensive agent time, companies look to IVRs, where conventional systems are a dime a dozen with a wide variety of costs and correspondingly low customer service ratings. Even million dollar IVR implementations offer no significantly different service levels.

Into the fray comes artificial intelligence and with it, a completely new business model.  Virtual Agents, powered by AI, are head and shoulders above other IVRs in functionality, ease of use, and ease of implementation. This technology has redefined how IVRs ought to perform, shifting the meaning of the “I” in IVR from “interactive” to “intelligent.”

Balancing Customer Experience and Cost-Savings
As a CFO, you look at budgeting, keeping costs in line, and making sure that you don’t cut costs so much that it will affect customer experience and your company’s reputation. As a new model of intelligent IVR has evolved, so has a new breed of CFOs: ones that can significantly add to the bottom line of a company and simultaneously enhance its services. The challenge is no longer just dollars and cents, but dollars and cents while bettering customer experience and elevating customer satisfaction. In other words, CFOs become responsible for more than just the financial health of the company. They too need to be concerned about the health of the company’s relationship with customers, as well as the health of the company’s financial outlook. One of the most visible and impactful ways for a company to improve both is by implementing an intelligent voice system.

The most important cost-savings from a truly intelligent IVR will come from a reduction in live agent time. A CFO has already done a cost analysis that breaks down how much each agent costs per minute of live time, and it is likely well above $0.50 per minute. That per minute price typically includes the hourly rate, HR benefits, and equipment like computer, phone, headset, etc. But there are huge indirect costs that may not have been included in that determination: training costs, Quality Assurance, Workforce Management, floor supervisors, manager salary and benefits, and more. All in all, many call centers end up with an average cost-per-call price tag in excess of $5.00, and sometimes, three, four, and five times that! And maybe that seems worth it – your agents are highly trained and efficiently used. But that same CFO would have to listen if there were a buffer between customers and agents that could respond to complex inquiries and came in cheaper than $0.50/minute.

For more information about how our system does that, click here.

Today’s customers are tech savvy, time conscious, and know what they want. An automated customer service system that understands that will retain customers, drive brand loyalty, and ultimately help the bottom line. The CFO that understands that will emerge as a visionary, one that is at the forefront of technology and a driver of company performance.

About the Author: Mike Vanca, Senior Vice President, Operations

Mike has over 30 years of experience in operations, IT, and corporate governance. At SmartAction, he is responsible for customer satisfaction and manages client development projects – everything from scope development, engineering, technical support, quality assurance, and ongoing maintenance.

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


Tropo includes US English speech-to-text transcription directly in the API. To transcribe any recording, simply set the transcriptionOutURI parameter on any recording and Tropo will transcribe the recording and send the transcription to that URL.

What if you want more capabilities? Other languages, tunable accuracy, human-powered transcription, or voice analytics, for example. For this, we can send your recording off to any transcription service that allows an audio upload. In this example, we’ll show how to integrate Tropo with Voicebase’s audio indexing and transcription API.

VoicebaseVoicebase provides an API that can transcribe in multiple languages, provides fast, accurate transcription for longer-form text, and can classify and analyze the resulting transcription. At the basic level, you can simply get a transcription back. The audio file and transcription are then stored in your Voicebase account for searching and detailed analytics. Mention Tropo when you sign up for Voicebase, and they’ll transcribe 200 hours of recordings for you for free.

A Tropo recording file gets sent to a URL of your choice. To send this to Voicebase, you’ll need a small application that receives the Tropo upload and then creates the Voicebase API call to send the recording for transcription. The application then waits for the transcription to be completed and does something with it, perhaps emailing it, storing in a database, or sending a text message.

The sample application described below uses Slim Framework, Tropo, and Voicebase and will run on any PHP web server. You can get the entire app in our voicebase-php repo on Github. It receives the Tropo recording and saves it to disk. It then asks for a basic machine transcription from Voicebase, and once Voicebase has finished transcribing the file, places the transcription in a text file with a filename that matches the audio file name. The application is less than 100 commented lines of code.

The building blocks

Tropo uploads the file to your web server using HTTP POST, sending the file in a field called “filename”. In PHP, you can access this file with the variable $_FILES['filename']. This file is going to be saved in a place the web server can serve it up later.

The Voicebase API uses the same URI for all API calls, and an “action” parameter in the API call tells Voicebase what API call is being made. All API calls are authenticated with an API key and the Voicebase account password, also both passed as API parameters. A version parameter indicates which VoiceBase API version is being accessed.

To upload a recording, a POST request will be used. The API name is uploadMedia. Instead of sending the recording file to Voicebase, the API call includes a URL where Voicebase will download the recording. And the Voicebase API allows an externalID to be set that later be used retrieve the transcription.

Transcription is asynchronous. The upload API call returns immediately, and the transcription is fetched with another API call when it is complete. The upload API allows a callback URL to be set, and when the transcription is complete, Voicebase will send a webhook request to this URL.

The webhook is a GET request with a series of query string parameters. The two that we’re interested in are state, which indicates whether or not the transcription worked, and externalId, which contains the unique ID we set for this recording.

To fetch the recording, a GET with query string parameters is sent. The API name is getTranscript. The version and authentication keys are included in the query string. The externalID is specified in the query string to tell Voicebase which transcription to retrieve. The format parameter indicates how you would like the transcription formatted. Voicebase supports a variety of output formats including plain text and even PDF.

The Tropo application

In your Tropo application, use one of the recording methods to make a recording and set the recordURI to http://your-server.com/path/to/application/recording/{uniqueid}, replacing your-server.com with the hostname of your server, path/to/application with the directory where you installed the application, and {uniqueid} with an ID that will be unique per call. This unique ID will be used as the Voicebase externalId and will also be used as the filenames for saving the recordings and transcriptions.

Generating a unique ID can be as simple as using the timestamp and caller’s phone number, like so:

<?php
$id = $currentCall->callerID . '-' . date('Y-m-d-His');

record('Leave your message',
    array(
        'recordURI' => 'http://your-server.com/path/to/application/recording/' . $id
        )
    );
?>

Application Walkthrough

In the Slim application on your web server, the function that accepts the recording upload from Tropo and then sends it to the Voicebase API looks like this:

$app->post('/recording/:id', function($id) use($app) {
    $dir = getcwd();
    move_uploaded_file($_FILES['filename']['tmp_name'], "$dir/audio/$id.wav");
    $app->log->debug("SAVE $id / $dir/audio/$id.wav");

    $params = array(
        "version" => "1.1",
        "apikey" => $app->config('apikey'),
        "password" => $app->config('password'),
        "action" => "uploadMedia",
        "transcriptType" => "machine-best",
        "mediaURL" => $app->request()->getUrl() . $app->request()->getScriptName() . "/audio/$id.wav",
        "machineReadyCallBack" => $app->request()->getUrl() . $app->request()->getScriptName() . "/transcription",
        "speakerChannelFlag" =>  'true',
        "speakerNames" => 'speaker-1,speaker-2',
        "externalID" => $id
    );
    $response = Requests::post("{$app->config('endpoint')}", array(), $params);
    $app->log->debug("UPLOAD $id / " . $response->body);
    if ('SUCCESS' != json_decode($response->body)->requestStatus) {
        $app->log->error("UPLOAD $id / " . json_decode($response->body)->statusMessage);
    }
});

Lines 36-38 accept the Tropo recording as a form post upload and saves it to the audio directory, using the unique ID created by the Tropo script.

Lines 39-50 are creating the data that will be sent to the Voicebase API. Lines 41 and 42 set your Voicebase account credentials from the configuration file. Line 43 instructs Voicebase that we’re using their uploadMedia method to send a recording, and line 45 gives the URL on your web server where Voicebase can download the recording from. Line 46 sets a webhook that Voicebase will hit when the transcription is complete, and line 49 sets an ID that we can use to locate the transcription using Voicebase’s API.

Line 44 sets the transcription to use “machine-best” to get the highest quality transcription available. This is only available for US English recordings, so if another language is used, you should omit this line. The next version of the Voicebase API will default all transcriptions to the highest available quality.

Lines 47 and 48 tell Voicebase that we’d like to separate different speakers into different transcriptions. Tropo records all multi-party calls in stereo, with one channel being dedicated to the first leg of the call and the other channel containing the other speakers in the call. For a transfered call, this results in both people on the call being recorded in different channels. For a conference call, this allows you to record each caller in the conference call and in each recording, one channel will contain only that caller. Voicebase will transcribe each channel separately and will tag each line of dialog with the speaker names you specify.

Line 51 makes the HTTP POST to the Voicebase API.

When Voicebase has completed the transcription of the file, they will make a request to the webhook callback we supplied with our recording upload API call. The code to handle that webhook request, fetch the transcription, and save it is below:

$app->post('/transcription', function() use($app)  {
    $req = $app->request();
    $state = $req->params('state');
    $id = $req->params('externalId');
    $app->log->debug("CALLBACK $id / " . json_encode($req->params()));

    if ($state != 'MACHINEREADY') {
        $app->log->error("TRANSCRIBE $id / Error in callback: $state. " . json_encode($req->params()));
    } else {
        $params = array(
            "version" => "1.1",
            "apikey" => $app->config('apikey'),
            "password" => $app->config('password'),
            "action" => "getTranscript",
            "format" => "TXT",
            "externalId" => $id
        );
        $qs = '';
        foreach ($params as $k => $v) {
            $k = urlencode($k);
            $v = urlencode($v);
            $qs .= "$k=$v&";
        }
        $app->log->debug("REQ TRANSCRIPT $id / {$app->config('endpoint')}?$qs");

        $response = Requests::get("{$app->config('endpoint')}?$qs", array(), $params);
        $app->log->debug("TRANSCRIPT $id / " . $response->body);

        $transcript = json_decode($response->body)->transcript;
        $dir = getcwd();

        $file = fopen("$dir/audio/$id.txt","w");
        fwrite($file,$transcript);
        fclose($file);
        $app->log->info("transcribed $id / $transcript");
    }
});

The webhook callback that we get from Voicebase has a couple of query string parameters in it. A state parameter tells us what the status of the transcription is. Did it succeed? This will always show that it was a success, since the only callback we asked for when setting up the API on line 46 was for successful transcriptions. A externalId parameter contains the unique ID that we sent with the API call on line 49. These query string parameters are extracted in lines 59-61.

Lines 67-74 set up the Voicebase API call that will get the transcription. Lines 69 and 70 provide your Voicebase credentials. Line 71 says we want to use the getTranscription method. Line 72 specifies that we want the transcription as plain text. Line 73 is our unique ID for the transcription so that Voicebase returns the right one.

The transcription format asked for here is plain text. Voicebase also supports other formats, including a JSON body that includes a timestamp for every transcribed word. For simplicity’s sake, we want a plain text result, so the Voicebase API JSON response contain the transcription as text in a single JSON property.

Lines 76-80 convert the data hash from lines 67-74 into a query string. Line 83 makes the API request to Voicebase and line 86 extracts the transcript from the API response. Lines 87-91 save the transcript as a text file using the same filename format as the recording was saved in.

Installing

First, clone or fork our voicebase-php repo on Github.

To test this application out, you’ll need a webserver running PHP. You’ll also need to get an API account from Voicebase. Mention Tropo when asking for your API key and Voicebase will give you 200 hours of transcriptions for free.

Rename sample.config.json to config.json and edit to add your Voicebase API key and password. Copy config.json. .htaccess and index.php to your web server. On your web server, use Composer to install the dependancies Slim Framework and Requests. Create a directory called audio in the same location as index.php and make sure it is writable by the web server.

The post Advanced transcription and analytics with Voicebase and Tropo appeared first on Tropo.

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


June 23, 2015

So, Signal happened. We’re pretty sure it happened because there are a ton of videos to prove that fact. It was a blur, a glorious blur. Amidst that blur, Matt Makai and Kevin Whinnery were running around Fort Mason herding cats, pushing out blog posts and assembling hack packs. In all the activity we missed a few talks. Here are the talks we wished we saw in person, and exactly what we were doing while the talk was happening.

[Watch all the videos from Signal here]

Kevin Whinnery – The Fox Loves Apparently Says It Loves HTTP APIs in Swift

One of the most popular sessions at Signal, but one I wasn’t able to attend, was “How Twilio Builds Twilio”. As a developer working at Twilio, but not directly on the infrastructure that powers the service, I am constantly awed at the ability of my co-workers to transform an HTTP request I send to their servers into an MMS message or phone call. In this session, our engineering VP Ott Kaukver leads a group of Twilio engineers in a session describing how engineering teams get work done at Twilio. It’s rare to interact with engineering leaders operating REST APIs and developer tools at Twilio’s level of scale, so I think this would be a great session for any person in a leadership role in their engineering organization.


What does the fox say? Apparently stuff about using HTTP APIs in Swift! All of my iOS native programming experience to date has come in Objective-C, so I was very excited to take in Leah Culver’s talk on using HTTP APIs from Swift. I was looking forward to seeing how to implement something practical, rather than the pure syntax stuff I had done in Playgrounds in Xcode previously. Leah is a skilled technical speaker whose talks I have enjoyed in the past, so I would definitely recommend her Signal talk to anyone looking to dabble in Apple’s new programming language.

Matt Makai – Just How The Hell Does Netflix Scale


I’m bummed because Netflix has open sourced a bunch of well designed and written open source tools for performing deployments on Amazon Web Services (AWS). Netflix is likely one of the top, if not the single largest user of AWS so their tools are built with many years of experience in that platform-as-a-service environment.

Mike even demoed the tools in action on the command line, going further than just talking about them through logos on slides. Seeing an example end-to-end deployment with these tools was super helpful compared to just learning about the tools in isolation.

I was running the Equip track so I missed every Insight track talk while watching every Equip track talk. Poor me.

Fear not, reader. If you suffered the same fate as Kevin and Matt and missed the Signal talks, you can re-watch them here.

The Signal Talks We Wish We Saw In Person

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


Earlier this year I wrote about sending and receiving Tweets using the Twilio and Twitter API. It turned out that I published the post on the same day as this article in the Washington Post on veteran broadcaster Larry King. Ricky, my buddy and fellow developer evangelist, sent me the link because of the following description of how Larry King sends Tweets.

When Larry King wants to tweet, he doesn’t log onto the Internet. He pops open the flip phone stored in the shirt pocket between his suspender straps and calls the number for a voicemail set up specifically for this purpose. Then he dictates a thought that will be picked up by an assistant and transcribed onto his @KingsThings Twitter account. And nearly 2.6 million followers are there to receive it.

This burned away in the back of my mind for a while until eventually inspiration hit. What if everyone could have an assistant to send Tweets for them? Sadly, that would cost quite a lot. Larry King can afford it of course, but could we build this service for everyone? If you’re thinking that this is something we can solve with Twilio, then you’re right! Twilio can record calls, like a voicemail system, and then transcribe them. Let’s see how we can add this to our existing application.

From SMS to Voice

As part of the series on Tweeting by SMS, we built up an application that could send Tweets when we sent a text message and send Twitter notifications back via SMS. We could extend this application once more to let us send Tweets from a voice call. To do this, we’ll need a few things:

If you already have the Twitter SMS application setup then jump on down to the next part where we start to implement this. If you’re just getting started, then clone the app from GitHub and we’ll walk through where to enter your credentials.

$ git clone https://github.com/philnash/twitter-sms.git
$ cd twitter-sms

To enter your credentials you need to copy the

config/env.yml.example
 file to config/env.yml.

$ cp config/env.yml.example config/env.yml

Then grab your Twilio Account SID, Auth Token and phone number, your Twitter application credentials and your own phone number into the spaces in

config/env.yml
. If you need to find out how to create any of those credentials, this blog post has all the answers.

Once you have everything set in your

config/env.yml
 file, install the dependencies with bundler and start the app to make sure everything is working.

$ bundle install
$ bundle exec thin start -R config.ru

Now we’re all set up, let’s set off down the road to tweeting like Larry King and build our own Twitter voicemail assistant.

Phoning it in

So far the application has two endpoints and some other features:

  • /messsages
     receives SMS messages sent to your Twilio number and sends them out as Tweets
  • /heath
    , a simple health check for a monitoring service
  • the
    twilio
     method, a shortcut to an authenticated Twilio REST client
  • the
    twitter
     method, a shortcut to an authenticated Twitter REST client
  • a Twitter streaming client, which listens for notifications from Twitter and sends SMS messages to your phone number

In order to Tweet like Larry King, we’re going to use Twilio’s recording and transcribing facilities. When we phone our Twilio number we’re going to need a new endpoint so that we can tell Twilio to record and transcribe our message. We’ll also need an endpoint for when the transcription is complete and Twilio sends us the text.

Open up

app.rb
 and let’s build that new endpoint. We’ll call it
/voice
 and we need to return some TwiML. I’m going to build in a message to greet the caller and then the instruction to record the call.

# app.rb
post "/voice" do
  response = Twilio::TwiML::Response.new do |r|
    r.Say "Hello Mr King, please leave your Tweet after the beep", voice: "alice", language: "en-GB"
    r.Record maxLength: 30, transcribe: true, transcribeCallback: "/transcription"
  end
  response.to_xml
end

The

<Say>
 verb speaks the instructions to the caller. I’ve set the voice to Alice and the language to “en-GB” so that she speaks with a British accent. King might be a fan if I got it to tell jokes too.

The

<Record>
 verb then sets the parameters for the recording. I’m only going to record for 30 seconds, that’s enough time to speak a Tweet (and the transcription only works to a maximum of 2 minutes anyway). I’ve set transcribe to true and included a transcribeCallback to which Twilio will send an HTTP POST request when the transcription is done. Notably the transcription is done once the call has ended, so this callback comes asynchronously. If you don’t include the callback Twilio will still transcribe the message, but you will have to call to the REST API to retrieve the transcriptions.

One further point about the

<Record>
 verb, once recording is complete Twilio will make another HTTP request to your application to find out what to do next. You can set the endpoint to make that call to using an action attribute on the
<Record>
 element, but if you leave it blank Twilio will use the current endpoint. For this application this will keep recording messages in a loop until you hang up the call. This could be useful if you have many thoughts in a row that need to be tweeted, so we’ll leave it like this for now.

Let’s add the transcription callback too. Twilio sends the contents of the transcription in the body of the request. To test, we’ll just log out the contents of the text.

# app.rb
post "/transcription" do
  puts params["TranscriptionText"]
  200
end

Testing it out

In order to test all of this, we’ll need to start the application up and make it available publicly so that Twilio can send the webhooks through.

$ bundle exec thin start -R config.ru

The application should be running on localhost on port 3000. To make the application visible publicly I like to use ngrok. Follow the installation instructions on ngrok.com for your platform and then start an http tunnel to port 3000.

$ ./ngrok http 3000

Ngrok will show you the URL that is now forwarding to your localhost, grab that URL and head to your Twilio numbers dashboard. We need to set our phone number voice URL to point at this URL. Edit your phone number and enter the ngrok URL with the path

/voice
 into the Request URL field as shown below:

Enter your ngrok URL into the Request URL field for voice

With all that in place, we can test our recording and transcription out. Give your number a call, leave a message and then check out the server log to see the transcription arrive. If it all works, it should look a bit like this:

The log should read out the transcribed message that you leftSequence has been shortened

If that works, then all we have to do is connect the transcriptions endpoint up to Twitter and we’ll be Tweeting like Larry King in no time.

Sending to Twitter

Since we’re using the existing SMS to Twitter app, we have a helper method available that gets us access to an authorised Twitter client. All we need to do is call the update method on the client, with our transcribed text and we’re tweeting. To tweet more like Larry King, you could throw a hashtag in there. King likes to use #itsmy2cents.

# app.rb
post "/transcription" do
  twitter.update("#{params["TranscriptionText"]} #tweetlikelarryking")
  200
end

Restart your server, pop open the flip phone stored in the shirt pocket between your suspender straps call your Twilio number and dictate your thoughts. Now you’re on your way to amassing 2.6 million followers as you Tweet like Larry King.

I Tweeted like Larry King, it was not perfectOk, so my first go wasn’t perfect…

Do other things like Larry King too?

Now, thanks to the simple power of Twilio we can all Tweet like Larry King! I don’t know if he does other things by phoning up and leaving messages, but using this outline we could do a whole bunch of other tasks with voice and transcription. You could:

There are other things you could do with this Twitter example too, you could

  • Watch the limits, if you do manage to record more than 140 characters worth of text, your tweet is going to be too long right now.
  • Get contextual, with a bit more work to search for the word(s) “hash tag” or “hashtag” turn the next word into a hash tag
  • Get personal, look out for the word “at” and if it precedes one of your follower’s names turn it into a proper @mention
  • Relax, just sit back, Tweet and feel like you have your own personal assistant

You can check out the full code for this example in the GitHub repo. Let me know if you have any questions in the comments below or by email on @philnash when you #tweetlikelarryking.

You Too Can Tweet like Larry King

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


June 22, 2015

Slack’s pitch is pretty simple: It’s the one stop shop for your team’s communication needs. It integrates with everything project management app so it can be come the project management app for your project management apps. I’ll save you the “Yo dawg, I heard you like embedding apps in apps” bit and get down to it: James Thomas just built out Phonebot, a Slackbot you can use to make calls within Slack using Twilio, and IBM Watson’s text to speech.

Here’s how it works. When you call @Phonebot, you can initiate a Twilio Call with your team. Watson listens to the call in chunks after the audio is posted to the REST API in 5 second batches. Watson then translates the speech to text and posts it in Slack’s chat. Now, you can even track Conference calls in Slack.

Here’s the rundown of the hack, originally posted by James Thomas on his blog here.

Building Phonebot: Integrating Slack, Twilio and IBM Watson

Slack publishes numerous APIs for integrating custom services. These APIs provide everything from sending simple messages as Slackbot to creating a real-time messaging service.

Phonebot will listen to messages starting with @phonebot and which contain user commands e.g. dial, hangup. It will create new channel messages with the translated speech results along with status messages. Users can issue the following commands to control Phonebot.

@phonebot call PHONE_NUMBER <-- Dials the phone number
@phonebot say TEXT <-- Sends text as speech to the call 
@phonebot hangup <-- Ends the active call
@phonebot verbose {on|off}<-- Toggle verbose mode
@phonebot duration NUMBER <-- Set recording duration
@phonebot help <-- Show all commands usage information

We use the Incoming Webhooks API to post new channel messages and the Outgoing Webhook API to notify the application about custom channel commands.

Listening for custom commands

Creating a new Outgoing Webhook, messages from the registered channels which begin with the “@phonebot” prefix will be posted to HTTP URL for the IBM Bluemix application handling the incoming messages.

We can create Outgoing Webhooks for every channel we want to register Phonebot in.

For each registered channel, we need to allow Phonebot to post new messages.

phonebot2

Sending new channel messages

Incoming Webhooks provide an obfuscated HTTP URL that allows unauthenticated HTTP requests to create new channel messages. Creating a new Incoming obfuscated for each channel we are listening to will allow Phonebot to post responses.

Each Incoming Webhook URL will be passed to Phonebot application using configuration via environment variables.

phonebot3

Making Phone Calls

Twilio provides “telephony-as-a-service”, allowing applications to make telephone calls using a REST API.

Twilio has been made available on the IBM Bluemix platform. Binding this service to your application will provide the authentication credentials to use with the Twilio client library.

When users issue the “call” command with a phone number, the channel bot listening to user commands emits a custom event.

bot.on('call', function (number) {
  var phone = this.channels[channel].phone

  if (phone.call_active()) {
    bot.post('The line is busy, you have to hang up first...!')
    return
  }

  phone.call(number, this.base_url + '/' + channel)
})

Within the “phone” object, the “call” method triggers the following code.

this.client.makeCall({
  to: number,
  from: this.from,
  url: route
}, function (err, responseData) {
  if (err) {
    that.request_fail('Failed To Start Call: ' + number + '(' + route + ') ', err)
    return
  }

  that.request_success('New Call Started: ' + number + ' (' + route + '): ' + responseData.sid, responseData)
})

The URL parameter provides a HTTP URL which Twilio will use to POST updated call status information. HTTP responses from this location will tell Twilio how to handle the ongoing call, e.g. play an audio message, press the following digits, record phone line audio.

If the phone call connects successfully, we need the phone line audio stream to translate the speech into text. Unfortunately, Twilio does not support directly accessing the real-time audio stream. However, can record a batch of audio, i.e five seconds, and download the resulting file.

Therefore, we will tell Twilio to record a short section of audio and post the results back to our application. When this message is received, our response will contain the request to record another five seconds. This approach will provide a semi-realtime stream of phone call audio for processing.

Here is the code snippet to construct the TwiML response to record the audio snippet. Any channel messages that are queued for sending as speech will be added to the outgoing response.

twiml = new twilio.TwimlResponse()

// Do we have text to send down the active call?
if (this.outgoing.length) {
  var user_speech = this.outgoing.join(' ')
    this.outgoing = []
    twiml.say(user_speech)
}

twiml.record({playBeep: false, trim: 'do-not-trim', maxLength: this.defaults.duration, timeout: 60})

 

When we have the audio files containing the phone call audio, we can schedule these for translation with the IBM Watson Speech To Text service.

Translating Speech To Text

Using the IBM Watson Speech To Text service, we can simply transcribe phone calls by posting the audio file to the REST API. Using the client library handles making the actual API requests behind a simple JavaScript interface.

var params = {
  audio: fs.createReadStream(file_name),
  content_type: 'audio/l16; rate=16000'
}

this.speech_to_text.recognize(params, function (err, res) {
  if (err) {
    this.error(err)
    return
  }

  var result = res.results[res.result_index]
  if (result) {
    this.transcript = result.alternatives[0].transcript
    this.emit('available')
  } else {
    this.error('Missing speech recognition result.')
  }
})

 

Having previously handling converting the audio file from the format created by Twilio to that needed by the Watson API, we were able to reuse the translate.js class between projects.

This module relies on the SOX library being installed in the native runtime. We used a custom buildpack to support this.

Managing Translation Tasks

When a new Twilio message with audio recording details is received, we schedule a translation request. As this background task returns, the results are posted into the corresponding Slack channel.

If a translation request takes longer than expected, additional requests may be scheduled before the first has finished. We still want to maintain the order when posting new channel messages, even if later requests finishing translating first.

Using the async library, a single-worker queue is created to schedule the translation tasks.

Each time the phone object for a channel emits a ‘recording’ event, we start the translation request and post the worker to the channel queue.

phone.on('recording', function (location) {
  if (phone.defaults.verbose) {
    this.channels[channel].bot.post(':speech_balloon: _waiting for translation_')
  }
  var req = translate(this.watson, location)
  req.start()
  this.channels[channel].queue.push(req)
})

When a task reaches the front of the queue, the worker function is called to process the result.

If translation task has finished, we signal to the queue this task has completed. Otherwise, we wait for completion events being emitted.

var queue = async.queue(function (task, callback) {
  var done = function (message) {
    if (message) this.channels[channel].bot.post(':speech_balloon: ' + message)
    callback()
    return true
  }

  var process = function () {
    return done(task.transcript)
  }

  var failed = function () {
    return done(this.channels[channel].phone.defaults.verbose ? '_unable to recognise speech_' : '')
  }

  if (task.transcript && process()) return
  if (task.failed && failed()) return

  task.on('available', process)
  task.on('failed', failed)
}, 1)

Deploying Phonebot

Now we’ve finished the code, we can configure the application to deploy on the IBM Bluemix cloud platform.

Configuring Webhooks

Phonebot must be passed the configured incoming webhooks URLs, allowing it to send channel messages. Following the standard Platform-as-a-Service convention for passing configuration, we store the channel webhooks as environment variables.

Using the CF CLI, we run the following command to set up the local environment parameters.

$ cf cups slack_webhooks -p '{"channel_name":"incoming_webhook_url",...}'

 

Application Manifest

Application manifests configure deployment parameters for Cloud Foundry applications. Phonebot will need to be bound to Twilio, IBM watson and custom services, along with configuring the runtime environment.

---
applications:
- name: phonebot 
  memory: 256M 
  command: node app.js
  buildpack: https://github.com/jthomas/nodejs-buildpack.git
  services:
  - twilio
  - speech_to_text
  - slack_webhooks
declared-services:
  twilio:
    label: Twilio
    plan: 'user-provided'
  twilio:
    label: slack_webhooks
    plan: 'user-provided'
  speech_to_text:
    label: speech_to_text
    plan: free

 

…with this manifest, we can just use the cf push command to deploy our application!

Using Phonebot

Phonebot will post the following message to each channel successfully registered on startup.
phonebot_is_here

Users can issue @phonebot COMMAND messages to control phone calls directly from the slack channel.

For further information about the project, follow the project on Github. Upcoming features are listed in the issues page. Please feel free to ask for new features, report bugs and leave feedback on Github.

Making Phone Calls Within Slack Using IBM’s Watson Twilio: Phonebot Is Born

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


June 19, 2015

I sat on the bus in wonder, both scared and excited for what would lie ahead of me when the doors opened. Anticipation and uncertainty dwelled in the pit of my stomach. I felt like a little kid on the way to the first day of kindergarten, when in fact I was in my sophomore year at The College of New Jersey. This bus wasn’t taking me to school, but to the financial district of Detroit, Michigan where I was joining nearly 1000 like-minded students, many undoubtedly going through the same anxiety as I. This event would be far from the normal eight hour school day. Today there was only one class and it was 36 hours long. This was MHacks III and it was my first hackathon.

Hacking Time

FacePress

The FacePress logo.

My nerves quickly settled when I realized that hackathons are not a competition but a facility to learn and grow as a developer. Events like MHacks strive to build a community of students and encourage them to try something new and focus on doing their best rather than being the best. Hackathons provide all of the resources for students to turn their ideas into reality.

The hack that my team worked on became known as FacePress. Its purpose was to solve a common problem amongst hat-wearers: How do you figure out what your fitted hat size is without trying one on? Many people don’t know that fitted hats adhere to a strange sizing scheme, making them difficult to purchase online.

The solution soon became clear and only one simple instruction was required: roll your forehead left to right across the keyboard.

The app calculated the greatest distance between any pressed keys, thus measuring a small portion of the user’s head. This measurement made it possible to calculate the circumference of the head using a little bit of algebra and a scale factor. The hack was built using nothing more than JavaScript with a little bit of jQuery and some HTML. After picking a catchy domain name, FacePress was a respectable, finished product.

Lessons Learned

The hackathon ended with a “demo jam” in which judges evaluated all of the hacks made over the course of the weekend. At the end of the day, FacePress did not win any prizes or receive any special recognition, but everyone that tested the app was intrigued and delighted by the ridiculousness presented before them. I was impressed by the finished product and how much I learned while building it. The overall experience was more valuable than any cash prize.

I learned a valuable lesson at MHacks, one that should always be kept in mind at events like this:

A simple idea should be built using simple tools. Far too often developers overcomplicate their apps by trying to incorporate every new tool available.

The lifespan of the average hackathon hack is rarely longer than a weekend. My advice: avoid trying to bring machine learning, real-time communication, OCR, and Arduino into one monolithic hack. The well-known UNIX philosophy says to “do one thing and do it well.” My team adhered to this philosophy unknowingly and it lead to a unique and surprisingly functional finished product.

The sheer number of people and the crazy things that they build are enough to overwhelm both experienced and first-time hackathoners. Start small and understand that there is no need to reinvent the wheel in order to have a good time.

No child has ever entered kindergarten expecting to learn everything on day one. Still, I left MHacks feeling like I learned more in a weekend than I did in a semester at school.

Looking Forward

Brodan @ TwilioMHacks would be the first of many hackathons for me and the starting point in my transition from hacker to community member. My name is Christopher Hranj and this summer and I am joining the Twilio team as a Developer Evangelist intern. I am now fortunate enough to work on a team that makes every day feel like a hackathon. Everywhere I go becomes a classroom where I can both teach and learn.

Serving the developer community enables and inspires me to make a positive impact on the world. My goal is to impart the lessons that I learned at my first hackathon to both hackathon attendees and anyone around me learning and building software. MHacks empowered me with a desire to learn that grows stronger every day. My job is now to pass that desire onto others in the community.

If you have any questions about hackathons or about my experiences as a developer evangelist, feel free to reach out to me at @br0dan, or find me in the Bay Area.

Introducing Twilio Developer Evangelist Chris Hranj

Bookmark this post:
Ma.gnolia DiggIt! Del.icio.us Blinklist Yahoo Furl Technorati Simpy Spurl Reddit Google


Last updated: July 05, 2015 01:01 PM All times are UTC.
Powered by: Planet

Speech Connection and Logos are Trademark of Nu Echo Inc. Copyright 2006-2008.