MediaElement for Sound effects

In doing Silverlight Simon and friends and then trying to port the uber tile from Silverlight to WPF I found that for doing quick short sounds there was a limit in Silverlight and WPF to the preformance and bahavior you can expect. WPF's 'MediaElement' is just horride but even in Silverlight if I try to use the same media element to play 2 short 2 second clips with in 1 second you will get issues. I must say though the Silverlight team has done a bang up job on the media element. It is orders of magnitude better then the one in WPF.

As part of pulling bits out of the crossfader codebase to post on codeplex I built a new 'MediaElement' class (see http://Crossfader.codeplex.com/) that solves this problem with sound in Silverlight and WPF and makes my life just that much easier for building composite apps.

So Karim suggested something called channel support to solve this and in this control I implemented it. Basically what happens is that the usercontrol called 'MediaElement' creates a collection of MediaElements that it then can have events and flags for for tracking state and each of these is a 'channel'. When you call SetSource the control looks for the first available channel to assigns the source. this actually creates like a quing mechinism that allows me to rapidly play sound effects without loosing one or having to wait for the control to have its mediaelement in a ready state and I don't need to have an army of media elements.

Also in WPF and Silverlight the usage can be then the same where I call SetSource and pass in a local uri path like this:


Target1.SetSource(http://localhost:52076/tone3.mp3);

or

Target1.SetSource("tone3.mp3");

in this case make sure the item is a local 'resource' in the xap with the control. In any case check out on the crossfader codeplex project or the HackingSilverlight library.

Converting Silverlight Apps to WPF

Using Ariel's Simon design I built this game called Silverlight Simon. Jobi suggested making it a WPF app and then a surface version. I got it working and will do a post on it later but strictly speaking here this is how I overcame some of the issues going from a Silverlight user control to WPF.

The first thing I found out when I moved my code into Xaml was a number of things that WPF doesn't do that you can do in Silverlight.

1. there is NO VSM... at all.
2. there is no autoplay on media element.
3. no 2.5D (ie, all the plane projection stuff)
4. and all my animation calls to story boards didn't work.

When I first ported the code over I knew the VSM and 2.5d was not going to work so I started with ripping that out. the first thing that then poped up was all my animations calls. For example say I have a click event and I'm doing something like [animationname].Begin();

well that doesn't work as all my animations are 'resources' I played with this for a while and with some Dr. WPF medicine I found that I basically had to change all my animation issues to something like this:


Storyboard foobar = TopSimonGrid.FindResource("PrefsBtnPressedKey") as Storyboard;
foobar.Begin();

from their I got my animations working. I then removed my auto play on the media element and it would compile. but as soon as an event tried to fire a sound I got another error. I ended up having to add
LoadedBehavior="Manual"

that way my sound method started working. At this point you could play the game if you could get past the fact that all the styles that used the VSM looked like crap. I know there was this codeplex project called WPF Toolkit ( http://www.codeplex.com/wpf ) which included a prototype VSM (Visual State Manager). I pulled it down and tried it out. The only hard part was figuring out the right name space to get my vsm: working. as it turns out it is where you would expect if it was part of WPF kind of:

xmlns:vsm="clr-namespace:System.Windows;assembly=WPFToolkit"

With this mojo going on everything worked but my flip fx control. I couldn't find that any one get industrous enough to build the 2.5d stuff for WPF so I took the old animations from the SL2 fx control but put them in the vsm flip animations generic xaml so that it would use that instead of the 2.5d. I then had to change the content presentor to have the transforms needed and then poof like magic Silverlight simon worked like a champ.

For something extra I ripped out the WPF window chrome so Simon sits on the desktop and I found that my drop shadow on Simon was know on my desktop.

for those that are interested ripping out the chrome which is way easier then it used to be I had to add this to the window tag:

WindowStyle="None" ResizeMode="NoResize" AllowsTransparency="True" Background="Transparent"

from there you can pretty much do what you want.

ok ok ok, here is a little update. for Simon the above works fine... but for my uber tile project a few more issues came up... starting with dependency property registration sanity check....

So it is not so much a difference other then that the code that shouldn't work does in Silverlight 3 but blows up properly in WPF. take a look at this code:

public readonly DependencyProperty BackLink2TitleProperty = DependencyProperty.Register("BackLink2Title", typeof(string), typeof(MediaPlayer), new PropertyMetadata(new PropertyChangedCallback(OnBackLink2TitlePropertyChanged)));

in the named registration value you can't have two of these with the same value. Silverlight will currently over look that little indiscretion. WPF... not so much :)

Rounded Corners MediaElement

so I was working on a new media player for Crossfader and wanted the media element in the new wizbang super tile to be able to form fit as in have roundy corners... I swear I spent hours trying to make this work and I guess had a brain block on this and thanks to Jobi pointing out the obvious the solution looks like this:

<mediaelement width="0" name="ThisMediaElement" height="0" stretch="Uniform" source="http://localhost:36825/Seiryu.wmv" balance="5" volume="0" autoplay="True" downloadprogresschanged="ThisMediaElement_DownloadProgressChanged">
<rectangle width="Auto" name="ControlBack" fill="Black" radiusy="30" radiusx="30" height="Auto">
<rectangle width="Auto" radiusy="30" radiusx="30" height="Auto">
<rectangle.fill>
<videobrush stretch="Uniform" sourcename="ThisMediaElement">
</rectangle.fill>
<rectangle.stroke>
<lineargradientbrush endpoint="0.972438,1.14352" startpoint="0.972438,0.000225328">
<gradientstop offset="0" color="#FF848484">
<gradientstop offset="1" color="#FF000000">
</lineargradientbrush>
</rectangle.stroke>
</rectangle>

seemed like the obvious solution but some how missed it...

Scrubbing Timeline MediaElement

I wrote this sample of a timeline a year ago in Dec, and some one posted a link saying that the code didn't work. gasp code written against the 1.1 alpha doesn't work against the released v2.0??? oh my, how can that be. Personally I htink the expectation was a bit over the top. Anyway i wipped out the same used in the book and yes I have not tried it in some time but working 18 hours a day on BillG keynotes and other such nonsense kind of does that to you.

anyway I got the code out and walked through it since actually I'm checking all the samples currenty for the final review so I had todo it. In any case I was suprised to find that it works, Yes granted the in the article and post a year ago didn't expressly bind the events but seeing as the audience is developers and this is a blog called 'hacking' silverlight so I'm assuming that most of the readers are trained developers and in fact many are probably smarter then me so I would think they would be able to figure out what it means to add the events. The only other issue I found with the code was in fact when I wrote the article the play and pause methods were abstracted in the original which again was a simple thing but anyway I found the remark mean. and here is the same code recompiled on RTM SL 2.0 and it seems to work:

http://www.hackingsilverlight.net/samples/ScrubbingTimeline.zip

Dr. Dobb's Article on Silverlight Media Elements

An article by me. :)

http://www.ddj.com/windows/207800861

Full Screen Silverlight 2

So I"m porting my old silverlight bits to silverlight 2 and I noticed I didn't see much are doing this so I thought I would post this up here for everyone. So in this case I'm making a media player control that I can drop in where ever I need it but I want it to be able to go to full screen. In my class I create a method like this:


private void TopElement_OnFullScreenChanged(object sender, EventArgs e)
{
SizeUI();
}

right now this calls a method that mucks up my UI depending on control size so when it is in full screen it just uses all the realestate. In the class constructor I bind this method to the host like this:


Application.Current.Host.Content.FullScreenChanged += new EventHandler(TopElement_OnFullScreenChanged);
I then create a bit of xaml that points at this method:


private void Canvas_MouseLeftButtonUp(object sender, MouseButtonEventArgs e)
{
if( SLHost.Content.IsFullScreen )
{
SLHost.Content.IsFullScreen = false;
}
else
{
SLHost.Content.IsFullScreen = true;
}
}

which that actually is dependent on this being a private member:


private SilverlightHost SLHost = new SilverlightHost();

with that nice a simple bit of code I have full screen mode :)

Video Brush in 5 minutes or less

The Video brush is a great way to do some cool special effects. To make a video brush work on you have to have a named media element that you can bind to. That has a bound source. In the sample below we have a source. We have set the element to be muted but play and its opacity is 0 so no one sees it on screen. Then we add some text in a text block app and add a foreground element with a ‘video’ brush. The video brush element is bound to the media element by using the media element name. Using a video brush you can basically make just about anything play video.

<MediaElement Name="MediaElementElement" Source="ImenSample.2.wmv" Canvas.Left="20" Canvas.Top="20" Width="100" Height="100" IsMuted="True" AutoPlay="True" Opacity="0" />

<TextBlock Name="Sample Text" FontSize="20" FontFamily="Verdana" FontWeight="Bold" Canvas.Left="10" Canvas.Top="50" Opacity="1" >PAINT WITH VIDEO <TextBlock.Foreground> <VideoBrush SourceName="MediaElementElement" /> </TextBlock.Foreground> </TextBlock>

When this Xaml is rendered we get the text ‘Paint with video’ that is actually playing the video in the background of the text.

Microsoft® Media Encoder™ in 5 minutes or less

Expression Media Encoder is a Microsoft tool in the Microsoft Expression Suite of tools specifically designed to take media content and either tanscode/encode it or produce in a form that can be consumed online in the form of some kind of Silverlight media player and WMV and to integration with Silverlight Streaming services. Expression media encoder has a number of built in templates for use in quickly building Silverlight media players. Once you get the application up and running, and you stare at it for a few minutes to get oriented it is pretty straight forward.

From the initial screen we need to import our media file. Go to the file menu and select import, navigate to the video file you want and selection ‘open’. Next we need to select a template which you can do by selecting output and then under the section ‘Job Output’ select one of the templates. You should see a shot from your video in the main area.

Now just click the button in the middle on the bottom and Encoder will go to town. Now this is all good and Media encoder has a bunch of other features around setting up markers and the like but the really key thing is that it gets video content into a nice format that Silverlight can consume.

http://www.microsoft.com/expression/products/overview.aspx?key=media

Silverlight Streaming in 5 minutes or less

Microsoft as part of the whole Silverlight ‘thing’ has provided a service to allow people to upload videos and get those video streamed along with providing other video streaming services. At the time of me writing this book it is a free service up to 4 gigs of space/video and streams video up to 700kpbs but as it goes into beta and is released you get advertising or you pay a ‘nominal’ fee but for developer’s and designers this is really cool and for companies this makes live a bit easier if you want to stream video content from you web site but don’t have a streaming media service.

A couple of key points of the streaming media service is to enable rapid application development, to be able to deliver quickly streaming media solutions and to be able to provide unlimited scaling should the need arise of your streaming content. The provide admin tools, a REST API, documentation and other tools online for your application development. Currently there are issues with Silverlight all living in the same domain so your bits out on Silverlight streaming need to be ‘all’ your bits but you can embed them in an iframe or separate window and pass in query string values etc. You can check out the service at :

http://silverlight.live.com/ or

http://www.microsoft.com/silverlight/streaming.aspx#1_4

So let us set up an account, starting from Silverligth.live.com.

Next we click on ‘get it free’. At some point a paid service will be available. But for now it is not.

Click the ‘click here’ link to go onto creating our streaming account.

If you don’t have a live id you will have to go through the registration process if you do then you can just login to create your account. In this case I do have such an id and I login and created my Streaming account and then the screen should look like this after logging in.
So when you click ‘I Accept, Create Account’ you then have your account.

This screen show your public account id and account key. You will want to know these later when we upload some content using the Microsoft Media Encoder. You can learn more about Silverlight Streaming at:

http://dev.live.com/silverlight/

So From here we are ready to for my next post on media encoder.

Video Scrubbing

A common feature of most video players is the ability to ‘scrub’ video or to ‘scrub’ the time line of a video. When we say ‘scrub’ we basically mean being able to change the position of the video by clicking on a line or icon and dragging it along a line that represents the video and the video changes in real time. That leads us into how we make the time line visually and then making it so we can ‘scrub’ it. The first thing we do is then is to create our Xaml. Check out our Xaml:

<Rectangle x:Name="TBStatusBarBg" Opacity="0.5" Width="327.167" Height="2" Fill="#FF8C8989" RadiusX="0" RadiusY="0" Canvas.Left="15" Canvas.Top="7.667" /><Rectangle x:Name="TBStatusBarDownloadProg" Opacity="0.5" Width="0" Height="2" Fill="#FF6C6B6B" RadiusX="0" RadiusY="0" Canvas.Left="15.333" Canvas.Top="8" /><Rectangle x:Name="TBStatusBarPlayProgress" Opacity="0.5" Width="0" Height="2" Fill="#FF353434" RadiusX="0" RadiusY="0" Canvas.Left="15" Canvas.Top="8.333" /><Rectangle x:Name="ScrubIcon" Opacity="1" Width="3" Height="8" Fill="#999999" RadiusX="5" RadiusY="5" Canvas.Left="15" Canvas.Top="6" Cursor="Hand" /><Rectangle x:Name="ScrubMask" Width="327.167" Opacity="0" Height="10" Fill="#FFF0E8E8" Canvas.Left="15" Canvas.Top="5" Cursor="Hand" />

You can make your time line any way you want but we will use 5 simple rectangles. So using our time line we will show how much the current video is downloaded and the current position in the play back. So with that our first rectangle is the background. This is the core visual representation of the video length. This doesn’t actually do anything and is static regardless of the video source but contrast of the progress and position indicator are relative to this on a percentage basis.

The next rectangle is used to show how much the video is downloaded. This rectangle is animated by an event off of the media element. The third rectangle is our position indicator that will show where the video is in playback and will in fact be the key thing that changes when we actually ‘scrub’ the video. Following that we have a rectangle that is used to as our indicators icon. It is rendered perpendicular to the others and is generally what a user will click on to scrub the time line. The last rectangle is actually an invisible mask we place over the whole sets that is actually what we bind mouse events to so that things are not so hard to click on.

The next two Xaml components are what we need to make the time line work. A media element and some story boards for animation. There are two story boards. One is a timer and the other is the position play back animation to smooth out the position change. Together this smoothes out the changes in position when scrubbing and during playback.

<Canvas.Resources> <Storyboard x:Name="Timer" Storyboard.TargetName="MediaPlayer" > <DoubleAnimation x:Name="TimerWidth" Storyboard.TargetProperty="Width" Duration="0:0:.1" /> </Storyboard> <Storyboard x:Name="PositionPlayBack" Storyboard.TargetName="TBStatusBarPlayProgress" > <DoubleAnimation x:Name="PositionPlayBackWidth" Storyboard.TargetProperty="Width" To="0" Duration="0:0:.1" /> <DoubleAnimation x:Name="PositionIconLeft" Storyboard.TargetName="ScrubIcon" Storyboard.TargetProperty="(Canvas.Left)" To="0" Duration="0:0:.1" /> </Storyboard> </Canvas.Resources> <MediaElement x:Name="MediaElement" Canvas.Left="0" Canvas.Top="0" Width="361" Height="280" ></MediaElement>


Now the hard part is actually in to wire things up. All the eventing we are doing programmatically but you could do it in Xaml as well. In our application object on load we create references to all the Xaml bits we will need to muck up. Then we bind events to the media element, to our animations and to the ‘scrub’ mask and then we set up the media element and we are complete in the onload. That key thing we want to keep in mind here is that we are starting up the timer animation. We could have this in a canvas trigger but doing it in code like this ensures that the time doesn’t start until everything is initialized and it does us some good.

this.TBStatusBarBg = rootElement.findName("TBStatusBarBg");
this.PositionPlayBackWidth = rootElement.findName("PositionPlayBackWidth");
this.PositionPlayBack = rootElement.findName("PositionPlayBack");
this.PositionIconLeft = rootElement.findName("PositionIconLeft");
this.Timer = rootElement.findName("Timer");
this.TBStatusBarPlayProgress = rootElement.findName("TBStatusBarPlayProgress");
this.ScrubIcon = rootElement.findName("ScrubIcon");
this.ScrubMask = rootElement.findName("ScrubMask");

this.MediaElement.addEventListener("DownloadProgressChanged", Silverlight.createDelegate(this, this.MediaElement_DownloadProgressChanged));
this.MediaElement.addEventListener("MediaOpened", Silverlight.createDelegate(this, this.MediaElement_MediaOpened));
this.Timer.addEventListener("Completed", Silverlight.createDelegate(this, this.Timer_Completed));
this.ScrubMask.addEventListener("MouseLeftButtonDown", Silverlight.createDelegate(this, this.ScrubMask_MouseLeftButtonDown));
this.ScrubMask.addEventListener("MouseMove", Silverlight.createDelegate(this, this.ScrubMask_MouseMove));
this.ScrubMask.addEventListener("MouseLeftButtonUp", Silverlight.createDelegate(this, this.ScrubMask_MouseLeftButtonUp));

this.MediaElement.SetValue( "AutoPlay", false );
this.MediaElement.SetValue( "Volume", this.DefaultVolume );
this.MediaElement.Source = this.Source;
this.Timer.begin();

The Timer then is only .1 seconds and then fires the completed event that we bound to it. In this event we check that the MediaOpened flag has been set and that that mouse is captured or not and starts the timer again. The media opened event that is bound to the media element sets the media opened flag when the video can start playing. Now if the media opened flag is set and the mouse is not captured then it calculates the width based on the width of the background of the time line vs the length of the video taken from the natural direction based on seconds and then the current position. Then we pause the current position animation and set the two values, one for the position bar and the other for the ‘icon’ or fourth rectangle we use as our scrub icon. After we set the critical values we then start the animation again so the change is smooth.

if( this.MediaOpened && !this.ScrubIconMouseCapture )
{
var Width = ( this.TBStatusBarBg.Width / this.MediaElement.NaturalDuration.Seconds ) * this.MediaElement.position.seconds;

this.PositionPlayBack.pause();

this.PositionPlayBackWidth.To = Width;
this.PositionIconLeft.To = Width + 15;

this.PositionPlayBack.begin();
}
this.Timer.begin();

So now we are ready to setup our scrub the time line. But first lets go over the download progress changed event. We bound this to show on our time line how much of the video is downloaded. So this event then sets the width of the download progress bar on our time line and then once it is downloaded it starts the media element playing if our property auto play we created in our class is set to true.

sender.findName("TBStatusBarDownloadProg").SetValue("Width", this.MediaElement.GetValue("DownloadProgress") * this.TBStatusBarBg.Width );

if( this.MediaElement.GetValue("DownloadProgress") == 1 )
{
if( this.AutoPlay )
{
this.Play();
}
}

A scrub operation then starts when a user clicks on our mask rectangle. This fires off the event Mouse Left Button Down. Here we first set the mouse capture flag, we call the pause method we created on the application class. We could just ‘pause’ the video our selves but since we might want to call this method in a number of places and in our application we want to change the play icon too we just wrap it in a method called pause which we call here. Once the flag is set and pause is executed we then actually ‘capture’ the mouse. Remembver on this method we loose the capture without being able to track it if the user moves off the Silverlight control. If we do nothing else if the user clicks some where on the time line to scrub then position will stay some where else until the move the mouse. So to make sure this weird effect doesn’t happy we also fire our mouse move event so that the position immediately goes to the correct position on the time line and the video play back.

ScrubMask_MouseLeftButtonDown: function(sender, eventArgs)
{
this.ScrubIconMouseCapture = true;
this.Pause();
sender.captureMouse();
this.ScrubMask_MouseMove(sender, eventArgs);
},

ScrubMask_MouseMove: function(sender, eventArgs)
{
if( this.ScrubIconMouseCapture )
{
var x = eventArgs.getPosition(null).x - 40

if( x > this.TBStatusBarBg.Width )
{
x = this.TBStatusBarBg.Width + 15;
}
else if( x < x =" 15;" to =" x" to =" x;" newtime =" x"> this.MediaElement.NaturalDuration.Seconds )
{
NewTime = this.MediaElement.NaturalDuration.Seconds;
}

try
{
this.MediaElement.position = "00:" + this.formatTime(NewTime);
}
catch(Exception)
{

}
}
},

The mouse move event as the user holds down the mouse button then actually mucks up our time line and video and is actually ‘scrubbing’ the video. First this method fires any time the mouse moves over the mask so as to not mess with the user too much we want to make sure we have the mouse capture before doing anything. If we do we then grab the x value of the mouse and change it by our offset of the control. Next we need to make sure the value is not to big or to small, for example if the user moves the mouse way past either end of the time line we would effectively be past the end or before the beginning so we reset our ‘x’ value that we will use accordingly using an if else if statement based on the bounds of the time line. Since the position change animation is current paused during the capture event because we called pause on mouse down we just set the two double animation values. Then we restart the animation. Also on regular mouse moves we just set the values and call the begin method. After this we then create a new time value conversion for what the new play back position will be. This is based on our x value divided by the line width divided by natural duration of the video. Then if the new time is greater then the natural duration we set it to the end of the media. Otherwise we go ahead and set the new position and use some error handling to make sure this operation doesn’t blow up.
When the user is done with scrubbing the video with our mouse move event and they lift the mouse button up we call our mouse up button to let the video continue playing. In this event we set our mouse capture flag back to false, call the play method which we wrapped so we could change icons etc like our pause we implemented and then we can release our mouse capture.

ScrubMask_MouseLeftButtonUp: function(sender, eventArgs)
{
this.ScrubIconMouseCapture = false;
this.Play();
sender.releaseMouseCapture();
},

You can implement this a number of ways but this is how I hacked it together and it works well. Now that you can scrub the video time line and your dangerous to yourself and others