333 replies to this topic

[tmek]

Today I've been continuing thinking about the idea of a universal event API.  Something that would be small, open source and community driven.  

 

It would be:

• Lightweight streaming protocol.
• Completely platform agnostic.
• Provides a way to communicate across 32-bit and 64-bit processes.
• Timestamps allow streams to be played back for debugging.

The question would be can the event types and their payloads be defined in a way that developers from enough projects could agree upon to make it worth defining.

 

 

A novel event type to add would be one that captures the ball position and velocity on every collision.  You could have full "replays" of games for both debugging and entertainment purposes.  I imagine some of these concepts already exist in VP or FP in some form or another and i'm just rediscovering the wheel, but this is more about doing it in a universal way.

Edited by tmek, 15 October 2015 - 03:58 AM.

[Brian Madden]

@Ben Logan (EDIT: I thought your name was "silver supporter" haha. ) - I haven't heard of DOF until now. And holy moly that's awesome! I actually laughed in awe watching that video.. like "hahahawowooow!" After Expo I'll really dig into it. (Great docs!) For sure if we get to the point where MPF machines are running on VP cabs then we should write a DOF module so we can talk to it. (If that's how it works.. I haven't read any of the docs yet so that statement may be comically ignorant.

 

@Tmek - So... this is really, really interesting. I don't know if you saw, but in MPF we have this protcol we invented called BCP for "Backbox Control Protocol." MPF is multi-process, with the core game engine which does game logic and controls the hardware, and then a media controller which handles the graphics, dots, LCD, and sounds. The two talk via BCP, a lightweight streaming protocol which is platform agnostic, can run between architectures, and allows timestamp recording so either side can be played back.

 

BCP game into existance because there were some people who wanted to do their graphics and sounds in Unity 3D, but they still wanted to have MPF drive the pinball machine. They reached out asking how that might be possible, and around the same time we were thinking of splitting MPF into two processes since we couldn't scale on small form factor computers that were multi-core, and so BCP was born. (And now with MPF, you can use our Python-based media controller or the Unity 3D one, whichever you prefer.)

 

I think we can take what BCP is today and make a "sister" protocol (do we call it PCP so the names match?) which is just like BCP except it sends all the pinball controller things we talked about. Here's the draft spec for BCP:

 

https://missionpinba...ng/bcp1-0-spec/

 

As you can see, it's a TCP socket which passes messages that are in the style of URLs, e.g. command?key1=value&key2=value

 

So I wonder if we can do something like that. Some examples:

• driver_pulse?number=2&ms=30
• driver_enable?numner=4
• set_light?number=2&brightness=255
• set_led?number=5&color=ff5500
• set_trigger?switch=1&driver=2&action=pulse&ms=30

(the set_trigger command is for programming those instant-response hardware rules I was talking about.)

 

And from the hardware to the game engine:

• switch_change?number=1&state=1&debounced=true

BCP also supports receiving DMD frames (since the frames are composed on the media controller but the pinball controller is what talks to the actual hardware. So when MPF wants to update a frame, the media controller sends:

• dmd_frame?data=<4096 raw bytes>

(4096 raw bytes for a 128x32 = 4096 pixels x 1 byte per pixel)

 

With PCP, the media controller could talk directly to the hardware interface instead of passing DMD frames to the pinball controller which turns around and sends them right back to the hardware.

 

We also are working on an addition to BCP for "screen space" LEDs which is where you can give all your LEDs or lights on the pinball machine x,y coordinates and then "play" videos rendered effects on them as if they were movie files, with each light or LED being tied to a pixel from the movie. (The movie is not visibly rendered, but it allows for crazy awesome light explosion effects and stuff.)

 

I don't know if this is the kind of thing you had in mind, but BCP has worked well for us so that's why I'm throwing it out there. Then we just have to make a PCP interface for MPF and we can talk to any hardware that uses it. Heck, maybe we'll rewrite our P-ROC and FAST platform interfaces so that they're just PCP providers.

 

When we first introduced BCP, some people were afraid it would be too slow. But since the TCP socket is held open and both processes are on the same computer, we see less than 1ms delay to get messages back and forth. And since the "instant response" stuff is handled by the hardware controller directly, even adding 5 or 10ms delay to trigger a sound effect or DMD slide is obviously not anything that any human will ever notice.

 

Also MPF already makes efficient use of the USB connection to the pinball controllers by sending all the commands in bulk at the end of a full game loop. So if the MPF game logic wants to fire a coil, it could be 5-10ms anyway before that command is flushed to the USB port. So I don't think adding a TCP socket in the mix will hurt anything.

 

Heck, it could even be faster because now we can use three processes for MPF (the hardware controller, the game logic, and the media controller), and on the little computers like RPi2 which are quad-core, now we can handle all hardware communication in a separate process.

 

Dang this is awesome.

 

So I really, really like this idea. BCP was not originally my idea and is a result of about 5 or 6 peoples' efforts, but most of them are here at the show. I'll talk to all them over the next few days and get their feedback. (I've already shared this thread with the extended MPF team which is about 10 guys now. We all talk via Slack.)

 

Man I'm stoked. I think I can put together an initial PCP spec pretty quickly. (EDIT: Just a first draft starting point that everyone can weigh in on.) )I can look at our platform driver interfaces and break out all the discrete commands they use. It's really not that many. 3-4 for drivers, a couple for switches, a couple for lights and LEDs, a couple for hw rules, a watchdog.. Heck, I think I could actually implement this thing in a day. It'd be interesting to test it on the real hardware side-by-side and see if any human can tell the difference. I bet not.

 

So I'm curious as to all of your thoughts on this? What do you think about a URL encoded TCP socket? (We can still strongly type the keys and values if that helps you. I'm a Python guy.. so to me "type" is whatever you want it to be. But of course the Unity 3D backbox controller already has a BCP interface. Maybe you can even use that code as a starting point for a PCP interface in Unreal? The Unity code is open source too: https://github.com/m...nity-bcp-server

 

EDIT: BCP uses TCP sockets, not UDP.

 

Exciting times!

Brian

Edited by Brian Madden, 15 October 2015 - 03:07 PM.

[tmek]

Heh, see here's where every developer is going to have their unique take on it.  But it's a good sign we both see potential in the idea.

 

I was leaning towards a lightweight and fast binary protocol.  In my test over named pipes I was able to have process "a" write an event, process "b" receives it and writes a response event and process "a" receives that.  All in 1/100th of a millisecond.  There's no parsing or decoding that has to take place you just read data from the stream and call a function or vice-versa.

 

My reasoning for why a binary protocol should be solidify-able is because the tech for older pinball machine doesn't really change.  New technologies get introduced into new or modified machines but that should translate into new event types being added to the protocol (such as RGB LED change or play media event in more modern machines).

 

This may be a case where fast enough is fast enough though.  Although, I recall in one of your articles a case were a driver was being started and then stopped very quickly like in a time frame of 1ms and then started up again 4ms later up to 200 times a second.  Was that a real world example?

 

I agree with all your potential application scenarios, I haven't been able to imagine anything that wouldn't be possible with something like PEP (not saying there isn't something, just that I haven't' thought of it yet).  I was also thinking of packing raw DMD frames in an event payload and even raw PCM audio samples if I can get them from the controller. Neither represent very large sets of data, but yeah video is probably where you have to draw the line and just have a play_media event that says, "load video x, start at time y and play for duration z."

 

Finally, how can you not love the name PEP for a speedy pinball virtual wiring protocol!  It's peppy!  

Edited by tmek, 15 October 2015 - 07:14 AM.

[toxie]

not that bad of an idea.

but at least for real machines, i don't know if the timing resolution/reaction time is good enough.

than on the other hand, if this is only driving another controller like PROC, that takes care of all the low level details/timings, it could be sufficient.

 

although i doubt the 'replay-a-game' possibility like described, as just storing collision information might not be good enough, as this would require that the physics engine is really really precise, robust and timing-invariant, which is usually not really the case (plus it might actually randomize things internally, depending on material properties). so there i guess one would need a complete sidepath to the engine and store ball-positions 'all the time' (e.g. discrete stamps plus interpolation between that) that are then just replayed (in addition to all the other events like switch triggers, etc).

[tmek]

although i doubt the 'replay-a-game' possibility like described, as just storing collision information might not be good enough, 

 

Thanks Toxie, That's true.  Perhaps for visual replays it could record a ball update event every render frame or at some regular interval.  I think though for debugging/troubleshooting you'd probably just be interested in the collisions and the events shortly after.

 

but at least for real machines, i don't know if the timing resolution/reaction time is good enough.

 

Two things that I've wondered about related to timing are:

 

#1 the timespan between pressing a flipper button and the flipper coil engaging.  

 

From some of the operations manuals I've read at least in some machines that is a direct connection. i.e. there is zero delay (apart from the speed of light).  Meaning, the MPU doesn't actually trigger the flipper coil in response to a button switch input it only enables the electrical path between the flipper button and the flipper coil when the game starts and disables it when the last ball drains and the game is over.  This might no longer be the case on modern machines with faster processors though.  I wonder in most real machines is there a delay, because it's going through the MPU and how much is that delay.

 

#2 The timespan from when a pinball rolls on a bumper switch plate until the bumper coil(s) are pulsed (same for a slingshot).

 

This is a case where the MPU is definitely firing a coil in response to a switch input. I'm curious how small that timespan can be on a real machine.

Edited by tmek, 15 October 2015 - 09:08 AM.

[toxie]

I think its not just the "reaction time", but rather having exact control over the electrical components. Unfortunately i know only the very basics about all of that (i'm more the software guy who can repair some basic pinball stuff when following instructions closely ), but remember from my short time with the FreeWPC project, that the CPU basically did all the control of the coils, etc, so if one wasn't realtime enough one could basically damage components, miss switches, etc. But i also honestly don't know if this is of any matter regarding the protocol you're designing.

 

So maybe you should rather contact somebody from that side of the pinball-world (like the PROC guys, Brian from FreeWPC, etc) to also give their comments.

[Brian Madden]

First, @Ben Logan, Apologies for thinking your handler was "Silver Supporter"! Please note that my handle is "Neophyte".

 

@Tmek, I cound get behind a binary protocol, but I would want to do some tests on binary versus URL encoded to see how much speed difference there is. I also want to take some measurements on MPF to see if any of these are faster or slower, and/or whether that matters.

 

That example of 1ms on / 4ms off is a real example (meant to "hold" coils at lower power so they don't burn up), however, MPF doesn't control the on/off at that level. That's sent down as a "pwm" command to the pinball controller (both P-ROC and FAST support this), so the command is like "turn on driver 4 for 100ms, and while it's on, patter it 1ms on / 4ms off."

 

One of the things I'm constantly amazed by is how "slow" physical pinball machines are relative to modern computers. Anything under 50ms is going to seem "instant" to humans. (I'm talking about the time because a target being hit and the light flashing, again not the "instant" rules to control pop bumpers and flippers.) So this may be one of those things where even if URL encoding is slower, it doesn't matter.

 

We might be able to combine elements of binary and URL encoded. For example to update 64 RGB LEDs, maybe the command is something like led_colors:?<192 raw bytes> instead of led1=ff5500&led2=ff5500&led3=556677...

 

Same for named pipes versus sockets. I don't have experience with named pipes, but that is also something to consider.

 

I admit that I like the name "PEP" better, but "PEP" in the Python world has a very specific meaning that is something else, so that might be confusing. But what we call this thing doesn't really matter, so if it ends up that people like PEP versus PCP verus PTCRPMFSW.. that's fine with me.

 

As for the capabilities of a pinball machine not changing over time, that's true, though new hardware will most likely be introduced that exposes new functionality that we'd need to include. For example, the P-ROC came out in 2009 and everyone saw that as the "standard," and then FAST came out earlier this year and added all sorts of new things. (Variable debounce times, separate debounce open and close times, variable coil recycle (i.e. "cool down") times, multi-stage pwm, etc. But as long as we have "room" in the protocol then that will give us the ability to evolve over time, so a binary protocol would be fine for that.

 

Actually the FAST serial protocol is a binary protocol. All of their commands are four bytes, and then all of the parameters are two bytes in a standard order. So really everything MPF sends to FAST is a 20-byte command. (Except for the DMD frames). Heck, we might be able to model after that as a starting point!

 

I'll ask the guys here about named pipes versus sockets and binary versus URL encoded just to collect some more points of view.

 

Bottom line though is yeah, I'm all for doing what the consensus is. No "sacred cows" here!

 

Brian

[Brian Madden]

Those other posts came in as I was typing my last response, so I'll answer them here.

 

The question about how "instant" the reaction time is.. that's a great question.

 

First, @Tmek is correct that the answer varies depending on the machine & era since they all used different technology. On "modern" (even 1990s WPC) machines it's purely CPU controlled (since modern CPUs are plenty fast enough), though the switch inputs are typically "direct" inputs which are serviced faster versus waiting for the switch matrix to be pollled.

 

Also another thing people don't think about is that physical switches have to be debounced, which means that they must be in the changed state for multiple simulatenous polls before the system considers the change valid. (This is to compensate for noise or jitter which might be enough to change the switch state for one random polling cycle here and there.) The amount of time the debounce cycle takes also varies.. sometimes it's 2 or 3ms, other times it's 10ms or higher. What's really crazy is that even the "instant" things, like a ball smacking a stand up target and bouncing off, we still see that standup target active for 20+ ms.. maybe times 50ms even! Like I said, pinball machines are "slow" compared to even the smallest form factor modern computers.

 

As others have pointed out, this is all fairly moot when writing new software to control physical pinball machines since the P-ROC or FAST boards handle the "instant" stuff locally. (With MPF, even diverters that need to trigger when an entrance switch is hit are handled by these hardware trigger rules.)

 

Brian Dominy (creator of FreeWPC and the developer for Heighway Pinball) worked with us on the initial architecture for MPF. (Though this was before we had BCP.) I know he's here at the show.. I'll ask him if he has any data about total turnaround times in the WPC era for switch hit-to-coil firing. (Or I can just look at the FreeWPC docs.. it's been awhile for me.) @Toxie @tmek are right that the in those days, the CPU was responsible for turning coils on and off.. and this was a 2MHz 6809 that was also responsible for polling the switch matrix, polling the direct switches, updating the lamp matrix, and composing the DMD frames! (In fact you will notice on real WPC machines that sometimes when a complex DMD show is taking place, the actual lamp outputs will stutter.) So the precision that Williams had in the 90s was far, far worse than what P-ROC and FAST can do today, and it was far worse than any overhead we'd have with some kind of PEP/PCP (whether binary or URL encoded, and named pipes versus TCP socket) protocol.

 

Obviously the ultimate test is just to prototype some things and try them out. But like I was saying, I suspect that the PCP/PEP thing might actually end up being *faster* than the way MPF does it now (and no one has ever complained about lag in MPF), especially since we can run it on its own core and use the USB bus in a much more efficient way.

 

BTW, when running a DMD-style game.. with full lights, DMD animations, sounds, etc., on my Windows VM on my laptop, MPF and it's media controller combined only use about 12% CPU, and that's with the current media controller that does all of its processing in the CPU. In November we will be moving to a new multimedia library that can use the GPU which should be even faster. We also have the option of compiling the slowest parts or our Python code into C (via something called "Cython") which can speed it up... something we'd probably want to do for a VP environment so we can leave as much room for the game simulator as possible. (Though I suspect that's GPU-bound and not CPU-bound?)

 

Brian

Edited by Brian Madden, 15 October 2015 - 06:52 PM.

[tmek]

The past few days I've been focused on "how" I might implement certain feature support in Pinball Labs.  I wanted to make sure I didn't lose sight of the exact features you guys were looking for while trying to come up with this catch-all plugin API.

 

I went through the thread and compiled this list of requested features.   Give a shout if there's anything I missed or something you think should be on here.

 

Requested Features for Pinball Labs so far

• Easy installation and setup
• Adjustment of table angle for difficulty/ball speed.
• Online leader boards per table.
• Adjustable Lighting
  • User adjustable Insert and GIstring lighting per table (LED vs incandescent bulbs, brightness etc.)
  • Option to toggle "Night Mod" / "Lights Out" toggle (dim/turn off environment lightings impact on the playfield surfaces)
  • Option to adjust environment lighting intensity based on lux value sampled from webcam or some other hardware. (auto day/night/lights-on/lights-off)
  • Option to adjust number of environment lights and their position/direction to approximate real world environment lighting.
• Alternative non-scripting ability to define game rules through the editor UI (scoring, mode progression, lighting, etc) 
• VB Script support
• VPinMAME support
• Simulation interface for pinball controller SDKs like MPF/pyprocgame
• Cabinet Mode features
  - Multi-monitor support: playfield, backglass, DMD
  - Adjustable, lockable playfield view per table.
  - Moveable, sizeable, rotateable backglass and DMD
  - PinDMD 1/2/3
  - Analog plungers: Nanotech DIY kit, VirtuaPin Plunger kit with accelerometer nudging
  - support for LED-wiz and equivalents (including Zebulon Booster boards)
  - configurable buttons
  - DirectOutputFramework support
  - Head Tracking plugin support

 

Some Questions:

1. Do PinDMD 1/2/3, LED-wiz and Zebulon Booster fall under DirectOutputFramework?  Or is there some other way they integrate with VP?
2. What all does the B2S Backglass Server do?  I noticed DirectOutputFramework seems to plugin through it in VP.  Can someone point me to some docs on B2S Backglass Server?

Edited by tmek, 22 October 2015 - 09:24 PM.

[Ben Logan]

@Brian -You are certainly welcome to call me Silver Supporter. Certainly more catchy than "Ben Logan!" Wonderful discussion between you, tmek, and toxie. I'm following it with interest (even though most of it is over my head, I'm getting a picture of where you guys are driving).

@tmek - I use DOF to power my LEDWiz. Edit: See Bent's info below regarding pindmd -- He knows his stuff!

Here's a great tutorial on how to install B2S, but I'm not sure it'll answer your questions regarding its architecture. Maybe a starting point (?)

http://www.vpforums....als&article=136

Edited by Ben Logan, 16 October 2015 - 12:06 AM.

[bent98]

 

Some Questions:

1. Do PinDMD 1/2/3, LED-wiz and Zebulon Booster fall under DirectOutputFramework?  Or is there some other way they integrate with VP?
2. What all does the B2S Backglass Server do?  I noticed DirectOutputFramework seems to plugin through it in VP.  Can someone point me to some docs on B2S Backglass Server?

pindmd is supported by vpinmame so if you support that it will work out of the box.

 

Ledwiz and Pacled 64 are boards that are used to send information to the pc via Direct Output framework. Since RGB LED's, Solenoids, contractors, fans, knockers and other toys draw current in the 1-4 amp range which the LEDwiz and Pacled cant handle (as they were designed mostly for single color traditional LEDs which are .1A range) Zeb came out with boards  that have  Octocouplers to take convert the high amperage and sometimes 24V to 12V and low amp so the LEDWiz and Pacled boards can handle them.  

 

Before DOF software someone came out with software that was called LEDwiz config tool that was used to create config files to work with the software that allowed you to configure each table to allows for LED lights and toys to map to the appropriate spot on your cab. Theres more to it then that buy I am trying to speak in general terms. The problem with the software is it would cause stutter as it created additional overhead and latency when running tables. Then came B2S backglass software which was a replacement for UVP Animated backglass which like ledwiz caused stutter. B2s solved the issue of stutter and allowed authors to created fully animated backglasses without the overhead. B2S had a plugin feature which would allow for others to levage B2s architecture to run additional functions. Swiss Lizard created DOF as a plugin to B2S. You can mostlikly add native support for DOF without having to run B2S if you plan is to add your own second screen support which is what I recommend. I suggest you work with SwissLizard directly. Hes a very nice guy based in EMEA.  Angrim and I worked on his beta team to bring it to the masses. There is a DOF Config tool that is very mature vs the old LEdwiz tool. I worked hard to get many configs up there. Over the years Argrim continued to evolve and improve the tools functionality and maintained the table configs as my life got busy and couldnt keep up with it. Its located on a site that is banned here and I can not mention it but if you goggle DOF config tool you will find it.

 

Edited by bent98, 16 October 2015 - 12:38 AM.

[zany]

Another nice thing would be some kind of online support. I guess in realtime is pretty hard to achive, but maybe the DMD or atleast the scores, for head to head play, or even full tournaments!

[tmek]

Another nice thing would be some kind of online support. I guess in realtime is pretty hard to achive, but maybe the DMD or atleast the scores, for head to head play, or even full tournaments!

 

Great ideas.  Online leader boards per table is something that should be possible early on.  The biggest issue there would be people who cheat and tamper with the scores sent to the server. 

 

Realtime multiplayer might be possible in future versions.  It would be awesome to take turns playing with a friend online with voice chat on the same machine just like you would in real life.  

 

If leader boards that got to be a big thing the replay system concept we were talking about the other day could become really interesting.  If you made top 10 on a table it would automatically upload that games replay for people to view.  But that's also something that would have to wait for future versions.

[zany]

True....some will probably try to cheat. No sure how to ensure that won't happend, but somekind of check sum, and also version look up on the table. I guess there is ways to atleast make it harder to cheat. Replays would be amazing, but i agree...first things first...but for future versions!

It would also be nice the ha some kind of challenge mode, like in TPA, where you have to go for different goals, and maybe an editor for that, where people can set up the goals for a table or a bunch of tables, and then share it.


...not TPA....i meant was Pinball Hall of Fame: The Williams Collection, don't think TPA have Challenge Mode.

Edited by zany, 16 October 2015 - 07:47 PM.

[Ben Logan]

tmek, 

 

I love that you are compiling a list of User Feature Requests. Just seeing that list made me smile. Thanks for listening. Happy Friday! 

 

 

[tmek]

I just finished a proof of concept for a Pinball Event Protocol server and client.  These are completely separate application instances.  All are hooked into a real-time pinball event stream with DMD frame events originating from the PEP Server which is running an instance of VPinMAME (though VPinMAME could just as well be hosted on one of the clients).

 

Each client is receiving the raw 128x32 DMD pixels and then drawing them to a 256x64 grid in a different color.

 

This also allows me to access the 32-bit VPinMAME from my 64-bit Unreal Engine development environment. 

 

Thanks to toxie for directing me to the 2.6 beta version of VPinMAME with his DMD output support.

 

Edited by tmek, 16 October 2015 - 10:09 PM.

[Ben Logan]

I know this isn't the intent, but that video views like a modern day Warhol piece. Love it! You're a wizard, tmek.

[tmek]

There's still quite a bit to do before this could be a playable table.  I mostly wanted to show some progress with the controller interface.  I find it satisfying as a computer geek that all this communication is going across a streaming event interface between multiple 64-bit and 32-bit processes and potentially hardware interfaces too.  If any of you dabble in electronic music this makes me think a lot of MIDI, but for pinball.

 

 

Here's an image showing how all the lights and switches will have their ID numbers hover over them when editing.  It makes it a lot easier to visualize lamp and switch locations.  Lamp IDs are white and switch IDs are green.

 

[Ben Logan]

That info graphic alone is suitable for framing!

And, FYI, I'm a fellow midi music / daw fan.

[Ben Logan]

Lighting looks "snappy" and responsive. I like that while this is a "day table" (as opposed to a "night mod"), the contrast between what's lit and what's unlit is totally clear. Nicely accomplished!

Not sure if you've tweaked them further, but physics look really good, too. And seeing that DMD up on the backglass is exciting.

In FishTales parlance, I'd say "You've got one on there!" tmek...

Edited by Ben Logan, 19 October 2015 - 10:15 PM.