Apple Siri. The Butlers are coming

Siri

Web2.0 democritised e-publishing and data creation in a friendly way for the masses and low and behold there are 182 million websites available on the net in 2011. Some of these websites have the lions share of the content (Facebook, Flickr, Google, Amazon etc) but collectively it's a grand publish of human thoughts, artefacts, wishes and desires.  What a wonder!

Creating content is one thing but leveraging insights across the content is more difficult. In truth there is still too much information for humans to effectively use and we find ourselves to be a gear in the machine rather than the driver - connecting systems together, cutting, pasting and rekeying.

I want to ask simple questions of my computers and have powerful background processing bring me the answer. Questions like "Which famous guitarists endorse products but don't use them in their live shows?" A query like this would require text analysis of the question to understand the meaning, scouring the net for famous guitarists,  checking which brands they claim to use in endorsements, checking their live 'kit' on websites, picture recognition of what guitars they are using, comparison of statements versus reality and then provide a weighted response based on the volume of data processed. Not easy and lots of key tapping.

Voice Control on the Bat Computer

It won't always be this way.

Batmans computer has been serving him for years (in the fictional world of DC Comics) controlled by his voice helping him fight crime. He simply asks the computer a question while he is driving or smashing heads of supervillans together and his Batcomputer gets back to him with the summary. Questions like  "Cross reference the known toxins that the Joker uses with chemical factories in the vicinity of Posion Ivy's locations over the past three months" are answered with ease. If a clarification is needed then it asks Batman. All achieved using natural language as the interface.

Digital buddies, assistants and advisors are here already for consumers albeit in the form mostly of recommendations engines and advertising systems. Last.fm helps reduce the millions of bands down to something I might like based on my previous listening while Amazon  advises me of books and products I might enjoy based on my previous activity.

These systems help us save time and slash the options and possibilities down to something we can handle. The volume of data falls below our eye and we can concentrate on the richer questions and answers.

For me the biggest aspect of the new iPhone 4S release was Siri - the virtual assistant. I think that as innocuous as it might appear on the surface (fixing calendars, looking up the weather, setting reminders) it is one of the first believable assistants that interact with consumers in a rich way.

Over time this service will grow to understand your accent, tone of voice and mood. It might voluntarily ask you what's wrong or question your commands if it thinks you are acting irrationally. It will potentially develop it's own personality and it will be answering more and more complex queries. Multiple Siris may even communicate and negotiate with one another to save their 'owners' from corresponding back and forth needlessly. Young children that can't type and older people may begin interacting with computers in richer ways. Siri may begin to find it's way into robots and other household devices outside of mobiles.

It's exciting and this is only the beginning. Others have tried to provide this kind of service but none have had the design and user base that Apple have in order to make it 'stick'.

I'll be watching this one carefully.

Stop Motion Animation on Mobile devices

I thought my recent foray into stop motion animation using mobile devices would be worth a few words. 

Timeline

Prehistory : A while ago I bought a cheap model of our solar system from a physics supply shop in South India. It was rusted and had been sitting there in the odd shop for ages. I shipped it back to the UK and it has been taking up space for a while without good reason. Recently, before moving house, I found a reprieve for it by deciding to use it in a stop motion music video. 

This blog is a look at how I got from that idea to a finished stop motion animation video and the software, hardware and people observations along the way.

Step 1 : Choose the track.

First things first was to choose the track - one I had in both English and Spanish - double search engine and geo-juice. Easy.

Step 2 : Storyboard the idea and the props

If you can't draw like your comic book heroes then a photo based approach to creating a storyboard is a good method. I wanted something on mobile that would let me quickly shoot, arrange and annotate the scenes. 

The unbeatable Cinemek Storyboard on iPhone does this job better than any other. I'm not aware of any compeition. You can just shoot and annotate anywhere meaning when inspiration strikes you move things along quickly. Projects in their early inception stage do depend on thinking done away from desks and desktops. The Cinemek UI is simple and storyboard arranging has a great 'physics' feel and touch making it  easy to swap the scenes around.

The screenshots below show how you can use it to 

• arrange
• camera movements against scenes  : tracking, pans, zooms, focus, lighting. Your still images can come to life.
• add notes and titles

I went and bought a new lamp, some special 'daylight' bulbs, black fabric, plastic bin bags, props, glue, wire and imported a iPhone4 stand from the US (Naja King). This stand allowed me to hang or stand the iPhone4 camera in all the different angles I imagined I would need. Word of note on the Naja- it can drive you mad with the slight movement it has once the new shape 'settles' thereby skewing the framing down after positioning responds to gravity. Naja has more flexibility and positioning capability than a Gorilla stand but less sturdiness - a trade off.

Naja King Flexible iPhone Stand

Step 3 :Choose Stop Motion apps and Prepare to Shoot!

Stop Motion Recorder

iTimeLapse

I must have bought and tried out all the stop motion apps that were in the Apple store.

The most fit for the job seemed to be Stop Motion Recorder and iTimeLapse.

What came out to be most important in the end was how reliable a program was. Losing 30 mins of work every 5 hours is not acceptable even if the application has all the whiz-bang features. iTimelapse wasn't reliable enough. After losing footage and periodic crashes and I resigned iTimeLapse to be a back up only. The trade off with using Stop Motion Recorder was that it was the 12 frames per second and low capture resolution. 

Step 4 : Shoot

People-ware : After doing a session or two on my own trying to get finished scenes I knew that an assistant with nimble hands could help me with the separation of shooting and set adjusting roles. Too much movement back and forth from camera to set takes up time and makes the whole thing slow. 

I hired someone I knew who studies fine art to help out with further set-making and the shoot. This was a wise move. 

We slogged it out and got most of the first half of the video. Stop motion animation is always more effort than you imagine. It's finicky! People say you should double estimates on average - with stop motion you treble it and budget for extra medication.

High Spec for
iTimelapse

After my time had expired with the assistant I had to shoot the remainder myself. Mostly the solar system scenes.  I noticed that iTimelapse had released a new version of their app and it was more stable. Quandary. Do I double the quality of image capture half way through the shoot or keep with the look/feel I have?

Using the logic that the video was in two halves 1) man in the house) 2) man in the solar system) I convinced myself that it could change resolution for the second half.

I looked more seriously into the audio trigger method on iTimelapse to take a photo i.e. shout "now!" then 'click' a shot is taken. It worked well once the sound levels were tweaked and even the misfires due to my thumping around in the 'scene' positioning things were easily edited out.

I managed to get the rest of the footage.

Step 4.5 - Make the decision about whether to continue developing the project solely on mobile. ..

This was quite a quick decision.

iMovie iOS (the major compositing tool on mobile for video) wasn't suitable because  :

• It has some memory problems and performance issues
• I simply needed a bigger screen to see what I had shot. It was time to take a good look.
• No ability to run plugins in-line and a lack of features
• I am more able with Final Cut Express than iMovie so pre-production with the mobile iMovie being imported into PC based iMovie wasn't a draw either

Maybe an iPad would have helped with the bigger screen but at the present time nothing beats editing and compositing on a powerful machine with a big screen.

Desktop and big screen win then.

Step 5 - Import

QuickTime import of Image Sequences

I imported the stop-motion data from my iPhone and quickly filled the iPhoto library up with thousands of photos only slightly different from one another (my wife took it well given it is a shared Mac). I spent some time fixing the orientation of some of the shots, due to the phone gyroscope flipping periodically, and then they were ready to use.

Final Cut Express does not do importing of individual frames to create video so I had to buy QuickTime Pro to get a good method of creating video footage from a sequence of photos. QuickTimePro also allowed me to specify the exact dimensions of the photos. The mobile apps had cut and trim problems when they exported video which is why I was working in photos and not video by this time.

Step 6 - Edit

Final Cut Express

Final Cut Express is a really good tool once you put some time in to learn it. My work with audio editing and compositing is translatable to video editing concepts so I had a quicker start when I first used it.

I have one of the KB Rubber Keyboard Covers that shows all the Final Cut Express shortcuts which is really useful. Shortcuts for me however are only a band aid as there are so many pieces of software I use that the only real solution for being quick with all my apps would be to have voice recognition.

I used only two plugins very sparingly - a contrast and brightness adjustment to make sure the plastic bin bags representing the cosmos looked dark enough and a plugin called  Lock and Load Express which smooths out shaky footage. Using Lock and Load has a tradeoff though as it selects a subset of the frame which bests stitches with the next frame. Using this approach it finds subset squares per frame as a path through the stills - like threading a needle almost. When shooting at 12 frames a second this was quite noticeable from a textural and resolution standpoint so I restrained to using it only for the manual stop-motion zoom effects I had tried which were very shaky.

Lock and Load Express

Step 7 - Master and Export


All done in Final Cut Pro. The export options are very simple and allow you to select high speed broadband as the likely consumption method making a QuickTime file small enough to upload to YouTube and other services.

I had an English and a Spanish version of the video so one export each while muting the other language track.

Step 8 - Promote

To get it indexed properly by the 'machine' I did the following :

• I used Spanish metadata for the Spanish language version whenever posted
• Put direct hyperlinks into the YouTube metadata pointing to my website
• Did a short music blog on Blogger and pinged feedburner.
• Put the video on Musician Profile Aggregator sites such as Artist Data, MusicSubmit
• I wrote this post to improve search engine links. This post is a technical article to create a cross domain link from science and technology into the arts and music
• Put it on my music website (both the front-page, videos section and on the album pages that it was taken from)
• Updated my YouTube channel to have this as the default video
• Put it on MySpace
• Did status update on my personal Facebook and my Facebook music page
• Put it on Twitter with hashtags to catch those searching for video and animation.

I could have done more submissions to other sites (Yahoo, MetaCafe) but I couldn't be bothered - YouTube, Google, Facebook and Twitter and the behemoths for video discovery. If need be I'll add more node juice later on.

I'll keep an eye on Google and YouTube Analytics for now to see how it gets on and whether the Spanish or the English version gets the most eyeballs.

Final

So is it possible to shoot and edit stop motion animation entirely on mobile? 

The short answer is that for preparation and capture that mobile is preferable but for editing and mastering the final video piece you still need a big machine.

Enjoy the video!

    

The final video is here in English. 

and here in Spanish

Wearable and Always On Computing

I'll be surprised if 2011 doesn't see something further happen around the wearable computing space.  We need to stop tinkering with metal boxes and facilitate direct interaction with the world a bit more.

There are two social dynamics to this kind of interfacing:

1. Broadcast the display externally on walls, tables, car bonnets or bodies (not private) OR
2. Broadcast internally on glasses or hidden earpieces (i.e. privately).

I think both approaches are more favourable to the current head down into a mobile neck stretch. Mobiles are private devices and Tablets/iPads a bit less so but they are both metal objects you have to put in front of your face and carry around.  The world is only there in periphery when using devices like these. 

Directly communicating with others and including the web as a 'third voice' is still not an elegant flow when taken out of presentation theatres and onto buses and high streets.

Pervasive and wearable computing will see an always-on environment for audio and video. The machines will listen to you 24/7 and parse what you say. The video components will continually record and pattern match the objects around you. Forget Amazon recommends when the data you can input is your whole day! We don't need to key the data about us like monkeys with typewriters. Spines everywhere will rejoice as we lift our heads to look back at the world once more.

The demos from MIT Wearable Computing Team in 2009 still look fantastic and the prototype only cost around $300 back then.



The TED talk - Pattie Maes' lab at MIT, spearheaded by Pranav Mistry



The interface ideas

The evolution of Steve Manns private eye glass display

BT FON: Now this is Social Computing

The premise of BT FON by British Telecom:

You freely allow for others to share a piece of your home-hub broadband connection in return for free access to theirs and, importantly, access for free to BTs public wifi spots.

It's potentially the worlds largest Wi-Fi community in the world and
my iPad, without a telcom data card, is begging me to join and download the iPhone app to activate it.

Those Wi-Fi hotspots were mostly aimed at businessmen but this seems much more democratising....and free. Let's be frank, using mobile data is definitely the easiest approach at the moment to feed your smartphone but it does seems we are at the stage now that those large towers used by telecoms companies to throw your data signals through the air will be replaced by a million peoples home wireless hub. It's social decentralised computing and the model is good.

I notice my existing mobile data provider, O2, has capped my mobile data usage and 10 days before the end of the month I find myself with it all used up and my speed throttled. If BT FON could take some of the load off then this would help somewhat.

I wonder what the telecomms companies will make of it?

I think this is an excellent play by British Telecom. If something like this could gain momentum then it would be quite a disrupter but without the numbers the experience will be poor as I transition between Bob Smiths hub and wait another minute until I can use a little bit of Mary's down the road. If switching between free hub-pimping and mobile data is seamless then maybe the problems aren't so great.

I'd be reading the small print on the security and privacy implications but this is definitiely one to watch.

A

Another angle on Social Computing - the iPad

There was a distinct ethereal hum as I finally added an iPad to my iPhone, iPod, Powerbook, iMac and G5 kingdom.

I had to answer some questions for myself on something that felt 'game changing' - magic almost.

What is the magic in the hardware, form factor and software of the iPad?

• Given it's size it travels from room to room with you. The only other computing device that really does that is a feature/modern phone. Laptops get parked which equals bad for a prospective command centre and true on body device.
• The role of phones will change. The size of displays for pocket devices have limits (technical and social) that the iPad immediately begins to highlight. Phone's may be more of a cousin of your car key, wallet and GPS devices than the final form factor of the utopian 'life controller'. Phone holographic technology would of course help phones reassert themselves as a social device rather than a private device.
• Works great as you walk and use the device using only 1 hand (unlike a laptop) for essential workflow tasks of the OS (cut copy-paste-search-open-close) , in fact most programs and tasks are possible given multi-touch to allow context. It also works really well with 2 hands allowing richer expression and depth of interaction.
• No wires and it fits everywhere - that's a fail for most laptops which demand height as well as depth on a desk or surface. iPad display and control are merged into one flat surface.
• The resolution is great - it looks like an interactive magazine.
• It wins the war hands down as the master input device for contacts and calendar control. Phone calendar co-working was not convenient with phones or laptops. Laptops feel 'official' and had to be crowded round with a bent neck (or worse via ping pong mail). The iPadCalendar experience is one where you co-author it with your spouse or friend or colleague sitting next to one another...on the couch, standing in the kitchen, and so on. Passing the iPad back and forth so everyone can download their details makes for good 'ownership' of events and to-do's and will prompt healthier use. This form factor, so far, is the best capture tool and it's social - just like calendars and are meant to be.
• The e-book, e-comic and e-magazine have arrived - they never had before. iPad allows you to zoom on article, photos, paintings or Maps. Hand it to a friend. Magazines can't compete...physical books also look like they will get in the neck strong if iBooks is anything to go by.
• Contact data and references (URL's etc) really are ubiquitous and synchronised across all my equipment. Retyping was hurting the mass uptake of computing - we need to do away with it. A combination of mobileMe, wireless and apple core products (itunes, iphoto et al) ensures that data, preferences and references are available, accessible and synced no matter where you are. No mean feat.
• Resolution of expression for multi-touch fingers. So much more satisfying for almost all application experiences from innocuous address book management and browsing to rich real-time control audio/visual applications.

The form factor of the iPad makes it a truly social device

The laptop and the mobile are personal computers - we don't share them physically with others much at all. They are social in that they can enable 'remotely social' experiences but it's a private affair.

This is where the iPad and form factors like it have a potential to shine ... families passing it round to arrange the trip to the lakes, band members trimming the email marketing list collectively, waiters allowing customers to select their choice and then taking the device to send wirelessly to the kitchen, putting in in grandma's lap to see slideshows.

but it's not all honey...

• The weight - any more weight for the iPad would be a fail but it just gets away with it.
• Heat - beware your iPad in strong sunlight...it heats up quickly and then forbids you to use it until it cools down.
• Power and Charging - Non native chargers that work for iPhones don't fare well with the iPad.
• Utilising other devices - I'm not a fan of the buy a wireless or a wireless with 3G simcard approach. It's a bad fit as iPad demographic probably already have mobile data contracts and don't want another. I'd have much rather payed an extra levy per month on my network provider bill (O2) to use my iPhone as a modem. The iPad is a natural main console for all your computing so it would have been nice to see utilisation of slave devices such as iPhone available out of the box - specifically from Apple rather than a 3rd party integrator.
• Upfront user profiles. Given the inherently social capability of the device it is a miss to not have controls for multiple user profiles.

We live in interesting times alright.

Mobile Widgets and Chips

Recently on a visit to a chip shop it struck me that chippie workers constantly have to say the word "chips" over and over and I had a chuckle at how they would slowly dement themselves with all the chip-based permutations day-in and day-out.

My laugh was short-lived though as the next day I caught myself saying the word "widget" at least 14 times an hour.

Widgets are the new chips alright.

Remember the old days when the media majors wanted you to go to their own portals and have your whole Internet experience from within their domain?




They all (MSN, Yahoo, AOL, BT et al) had this approach to try and hold onto the customer. It was the the shopping mall where they'd like to lock the door when you enter as when you left they couldn't successfully advertise, track and sell you things. This would be a fine approach if one web service could really offer us everything but clearly the world isn't like that on the web anymore.

I use hotmail, facebook, google (nearly all their applications), flickr, myspace and a tonne of other web services and before 'widgets' I would have to go to all these different sites running around to get to my data and services. Tabbed browsing helped make this a little easier but still, I was opening pages left right and centre.

Widgets, as a recap, allow you to take a chunk of a websties functionality and present it anywhere you would like - this allowed the data and service to come to you rather than the other way around. The graphical interface of your Flickr or YouTube widget has no dependency on Flickr or YouTube as the widget developer (and you) only want the data from these services. You can embed widgets in your Web2.0 profiles, your custom start pages or embed them in web pages you code yourself. They are modular little lego chunks of functionality that allow us to construct our own dashboards and environments.

Widgets are overground now and are used by bloggers, social network users, auction sites and owners of personal web sites. They exist on home page sites such as iGoogle, Netvibes, or Pageflakes.

Have a look at how I can easily aggregate widgets on one page to help with all aspects of my travel planning. A widget here, a widget there allows me to take the best of each web service and commoditise them to suit myself:




For the most part widgets are sandboxed and operate in ignorance of one another and mostly they manipulate data in the web cloud - it's just safer that way.

Recently there has been a rightful hoo-ha about mobile widgets i.e. widgets running on a mobile phone. For the most part web services on the mobile phone are consumed using a mobile browser or a dedicated application. The mobile browsers such as Safari, Opera Mini or the Nokia browsers are conceptually derived from their web counterparts and for the most part mobile web services via a browser, while improving, tend to be cumbersome with the best experience requiring multi-touch devices and continual zooming in and out to navigate pages that were essentially developed for the 'big screen' web.

The same principal that applied on web also applies to some widget platforms - bring the data to the customer rather than have them run around the web for it themselves.

The first big service in this space was by Nokia with their Widsets offering which was a Java program on your mobile that allowed you to add, view, and configure widgets directly from your mobile. To make life easier Nokia also provided a fixed line service to more easily allow users to manage how the service appeared on their mobile.


The market recently has exploded with widget offerings for mobile and it is in this field that I find myself caught up in - being earnest about widgets, looking at widget solutions, imagining the strategic importance of widgets - this ridiculous sounding little word now holds great gravity for me. I can no longer think of a 'mobile meal' without thinking "would you like some widgets with that?".

Mobile widget solutions have now moved on from standalone application style approaches, like Widgets, to richly integrated solutions that harmonise with a users idle screen (your phones dashbaord). The area is in transition and there are currently no standards on how to do this across multiple handset platforms and phone models.

The fight is on between Telecomms companies, handset manufacturers and large service providers.

Nokia has an S60 solution they are developing with the Symbian foundation, Opera has a widget solution that it is rolling out with operators, handset manufacturers are replacing their 'program menus' with richer widget style dashboards.

My dad would never use a mobile browser but I do believe he would use a Celtic football club widget alerting him of scores and news and providing him with simple links to video footage right from his mobile phone home screen. The simplicity and immediacy of web services has changed with these little critters called widgets and if you don't find yourself coming across this innocuous word alongside some market superlatives in 2009 then I, for one, would be very surprised.