Monday, April 21, 2014

Pyroscaphe: Adding a Touch Screen to PC Gaming

I have a confession to make.  I am a big fan of PC gaming.  Ok, that's not too controversial, but let me take this a step further.  I am a big fan of PC gaming...with a controller.

For all too long, the controller has been a second-class citizen in PC-land, with the mouse and keyboard reigning supreme.  This is why I was so excited when I heard news of the Steam controller that Valve is developing.  For those of you that don't know, this is a controller, currently under development, which is designed from the ground-up to work well with PC games, even those that traditionally could only be played with a keyboard and mouse.

Originally announced Steam controller design
The initial prototypes of the controller were impressive.  Obviously the dual trackpads are integral, but the part that excited me the most was the touch screen display at the center of the controller.  This touch screen was to be a multi-purpose input/output mechanism for games to utilize for auxiliary tasks or just as "soft buttons".

There are so many possibilities for a mechanism like this.
  • on-screen maps
  • inventory management
  • quick save/load buttons
  • ammo status for "realistic" shooter games
  • any auxiliary function that may not map well to existing controller inputs
To my disappointment, a few months ago Valve announced that the touch screen had been dropped from the steam controller design.  I'm sure they had good reasons to do this, but as an engineer excited by this concept, I just couldn't let this die.

Steam controller with touch screen removed

The Premise


Most of us already have powerful touchscreen devices in our pockets.  Rather than design a new controller with a built-in touch-screen, we could leverage the devices that we already have to enable these new types of game interactions.

That was my general line of thinking, and it's how Pyroscaphe was born.

Prototype Controller


Pyroscaphe


At its core, Pyroscaphe is a web server that allows you to interact with your gaming machine on a local network from a web-enabled device (like a phone or ipod touch).  This server serves web page UIs for a given game and also receives WebSocket requests which lets the player interact with the game in a real-time, low-latency manner.

In its current state, Pyroscaphe is mainly useful for sending keypress events to games, so through the UI on your mobile browser, you can essentially press keys on your keyboard.

In the future, I would like to make this more of a 2-way interaction, with the touch interface receiving more data about the running game.

Demo


(I apologize for the blair witch photography)



Why Web-Based?


There have been many articles recently that have declared the mobile web dead...or dieing, with the App being king.  However, for this particular application, I believe web-based UIs are best.

If something like this is to succeed, it depends heavily on the community generating content in an incremental, and distributed way.

This would be very challenging for a traditional mobile app.  With an app, we would have to have a central location for UIs to be reviewed and vetted, and they would have to go through the long Apple review process.  This process discourages frequent updates and also means that there must be a central authority that spends a lot of time hand-holding the process and vetting these touch UIs.

Since Pyroscaphe is essentially a locally-hosted web-app, it doesn't have these particular issues.  All of the content lives on the user's PC, where they have full control over what's installed and can update it incrementally with UIs created by the community.  There doesn't have to be any central authority.  UIs can be freely downloaded from ftp sites, discussion forums, or traded on floppy disk :)

Also, though inter-browser compatibility can be challenging at times, it's not too difficult to write HTML that works across most mobile browsers (as opposed to writing one interface in ObjectiveC, another in Java, etc..).

Finally, by using HTML, it greatly reduces the barrier to entry for creating one of these UIs.  HTML is a universal language that just about anyone can pick up fairly easily.

Client web page creation


Like I said previously, if Pyroscaphe is to have any chance to succeed, it must leverage the power of the community.  There are just so many talented and creative gamers out there.  If someone feels the itch to create a touch UI for their favorite game, I want it to be as easy as possible to do so.

One of the things I'm doing to accomplish this is to leverage HTML5 "data-" attributes.  Here is what my prototype "Duke Nukem 3D" page looks like:




<html>
  <head>
    <title>Duke Nukem 3D</title>
    <script src="/pyroscaphe.js"></script>
    <script> $(document).ready(function(){ initPyroscaphe(); });    </script>
    <style>
      <!-- CSS omitted for expository purposes -->
    </style>
  </head>
  <body>
    <table id="gunTbl">
      <tr>
        <td><img class="dukeGun" data-key="2" src="pistolIcon.png"></img></td>
        <td><img class="dukeGun" data-key="3" src="shotgunIcon.png"></img></td>
      </tr>
      <tr>
        <td><img class="dukeGun" data-key="4" src="chaingunIcon.png"></img></td>
        <td><img class="dukeGun" data-key="5" src="rpgIcon.png"></img></td>
      </tr>
    </table>
  </body>
</html>
As you can see, this HTML is extremely simple.  The bootstrapping mechanism is fairly lightweight as well.  Pages must pull in "pyroscaphe.js" and initialize Pyroscaphe when the document is loaded.

With that in place, pages can use the "data-key" attribute on an element to specify which key they want to press when a given element is touched.  In this case, weapons are selected with the number keys, so we bind the pistol image with key "2", shotgun with key "3", etc.

Behind the scenes Pyroscaphe uses jQuery to add the appropriate event handlers to send the right WebSocket commands to the server.

The prototype UI for Mortal Kombat is even simpler:



<html>
  <head>
    <title>Mortal Kombat</title>
    <script src="/pyroscaphe.js"></script>
    <script> $(document).ready(function(){ initPyroscaphe(); }); </script>
    <style>
      <!-- CSS omitted for expository purposes -->
    </style>
  </head>
  <body>
    <img id="finishHim" data-seq="V;A;L;V;E;" src="finishHim.png"></img>
  </body>
</html>
In this example, Pyroscaphe is bootstrapped just like before.  However, we just have a single image on the page with a "data-seq" attribute (as opposed to the "data-key" attribute that the previous page used).  As you've surely guessed, this tells Pyroscaphe to send a sequence of key presses to the game when this UI element is touched.

These are just 2 examples of very simple UIs that can be created.  Again, this is about leveraging the creativity of the gaming community to create these UIs.  The easier I can make it to create this content, the more likely it is that someone will do so.

Latency


Web apps have a bad rap for being slow and unresponsive, at least compared to native applications.  However, Pyroscaphe has several things going for it on this front.

It's designed to be used in a local network, so very few hops are required to get from client to server.  This also means interaction is more deterministic since we're not just sending data out into "the cloud", it's only going out on a small, controlled network.

We're also using WebSockets for all time-critical game interactions.  This means we don't have to deal with the overhead of creating a new socket connection for each server call (which itself involves a few round-trips).  We also don't have to deal with the overhead of the HTTP protocol.  As such Pyroscaphe has a very simple bare-bones command protocol that keeps performance snappy.

To validate that the latency wasn't too bad, I wrote a fairly simple test.


By round-trip time, I mean that this is the time it took for the UI to send a command to the server and for the server to send a simple response back.  So in reality, the lag between interacting with the touch UI and seeing a key press is roughly half of this time.  As you can see, in my tests the vast majority of calls had a round-trip time of 3-4 ms, which is more than sufficient for most game applications (and in-line with standard wireless controller latency).

A secondary benefit of using a single WebSocket (as opposed to many web-service calls with their own sockets) is that the data is serialized and we don't have to worry about data arriving at the server in a non-deterministic order.

FAQ


Won't the mobile web browser UI get in the way?


This can happen with some browsers.  I personally recommend using browsers that have a good fullscreen mode (like Atomic Web Browser Lite).

A touch screen on the controller diverts attention away from the action on-screen.  Won't that hurt the overall experience?


Touch screens are probably not the best input mechanism for really fast-paced action games.  However, there are a large number of games out there that don't require absolute non-blinking focus.  Those are the games that Pyroscaphe is intended for.

Can Pyroscaphe UIs send xbox 360 controller (XInput) button presses?


Not at the moment.  Currently, you have to use a program like Joy2Key to have your controller send key presses (and configure your game for keyboard input).

I would certainly like to get this working at some point.  However, I couldn't find an easy way to do this.  I could probably create a driver for this purpose (I personally work a lot with drivers by trade), but that's more trouble than I wanted to go through for the initial 0.1 version of Pyroscaphe.

Why are your demo UIs so ugly?


Sorry, UI design isn't my strength.  I'd be thrilled if someone wanted to help make these demos look better (or create new UIs).

So this can only send keypresses to a game?  So what.


It's not much right now, but I would like this to be the start of something better.  It's much easier to get keypresses to a game than to get data from a game (like showing an interactive map with your current location).  I suspect that doing the latter will require some game modding which isn't something I've done before.  Again, any help is welcomed.

Future Direction


There are a lot of possibilities for Pyroscaphe in the future.  Here are a few ideas I've had.

  • Auto-detect the game that has been launched and instruct the touch-screen to change UIs.
  • Simulate controller key presses via XInput and/or DirectInput
  • Simulate a mouse touchpad for navigating complex game menus
  • Detect key bindings or otherwise make it easier for users to map a given touch UI to their key bindings.

Conclusion


It's not much right now, but my hope is that if there are others like me out there, Pyroscaphe might gain some momentum and eventually lead to some really interesting and engaging gaming experiences.

Give It a Try!

Pyroscaphe's code is open source and freely available on GitHub.

https://github.com/justindvs/pyroscaphe

Don't want to build it from source?  You can download a binary distribution for Windows x64 here:

https://github.com/justindvs/pyroscaphe/releases

46 comments:

  1. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info. large touch screen monitor

    ReplyDelete
  2. I am unquestionably making the most of your site. You unquestionably have some extraordinary knowledge and incredible stories. Gamestorrents

    ReplyDelete
  3. Automatic Profile Switching is a feature that allows you to automatically or manually change the profile you are in or want to be in. triche fortnite

    ReplyDelete
  4. One thing is certain. Kids love them. They buy and play them in ever increasing numbers. Electronic games are here to stay.
    gta 5 modded

    ReplyDelete
  5. Moreover, it has one more option which really makes me to feel like I driving lively because it has vibration motor which indicate to lock the tires through this motors. Get Mini Militia

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. Its response is as faster as you one click of your finger tips to the keyboard and its reaction is on the screen. Moreover, it is easily adjustable and can move without any tensions.ashley

    ReplyDelete
  8. It is believed that this kind of practice would provide benefits in real life where you would be able react with speed and take quick decisions.
    https://donkeykongroms.com/donkey-kong-country-gbc-rom/

    ReplyDelete
  9. Just admiring your work and wondering how you managed this blog so well. It’s so remarkable that I can't afford to not go through this valuable information whenever I surf the internet! happyluke

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Wenn Sie nach unbegrenztem Geld und PR suchen, um Ihre GTA 5-Aktivität zu überprüfen, besuchen Sie meine Website : Gta 5 Geld Hack

    ReplyDelete
  12. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own BlogEngine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it. best gaming pc

    ReplyDelete
  13. Wenn Sie nach unbegrenztem Geld und PR suchen, um Ihre GTA 5-Aktivität zu überprüfen, besuchen Sie meine Website :
    Gta 5 Geld Hack
    Gta 5 Geld Cheat
    Gta 5 Geld Glitch

    ReplyDelete
  14. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article... Best Gaming Motherboard 2019:10 Hand-Picked

    ReplyDelete
  15. I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. https://www.buildbeasts.com/parts/case/

    ReplyDelete
  16. All of the content lives on the user's PC, where they have full control over what's installed and can update it incrementally with UIs created by the community. get more information

    ReplyDelete
  17. I am extremely delighted in for this web journal. Its a useful subject. It help me all that much to take care of a few issues. Its chance are so awesome and working style so rapid. get more info

    ReplyDelete
  18. This comment has been removed by the author.

    ReplyDelete
  19. This would be very challenging for a traditional mobile app. With an app, we would have to have a central location for UIs to be reviewed and vetted, and they would have to go through the long Apple review process. home page

    ReplyDelete
  20. There are also many games available in the market.

    ReplyDelete
  21. Regardless of whether a portion of these diversions can appear to be bland because of its realistic nature and savagery, they can play a positive and significant job in a youngster's improvement by advancing cooperation, building certainty and improving engine aptitudes. Read This reviews

    ReplyDelete
  22. I'm impressed, I must say. Very rarely do I come across a blog thats both informative and entertaining, and let me tell you, you ve hit the nail on the head. Your blog is important.. https://bestgamingthings.com/best-full-tower-case-for-gaming/

    ReplyDelete
  23. Quickly this site will indisputably be famous among all blogging people, because of its fastidious articles or reviews.
    watch it

    ReplyDelete
  24. Thanks a lot for your informative article.
    fortnite v bucks generator free v bucks hack v bucks

    free free v bucks generator, free v bucks ps4,free v bucks codes,

    v bucks gift card
    fortnite giveaway
    fortnite v bucks prices
    fortnite code free
    generate v bucks
    free fortnite redeem code
    fortnite v bucks free

    If You Need Fortnite V Bucks Free, Please Visit Following Links.

    Get Free Fornite V-Bucks Codes

    Get Free V-Bucks Generator

    ReplyDelete
  25. Fortnite Free V Bucks Generator will make you get unlimited free v bucks. Get most of the resources and in-app purchases free of charge, using Fortnite V Bucks Hack generator. Just insert the codes found on this site into the Fortnite V Bucks Hack generator tool. The Fortnite V Bucks Hack generator work fast and it is safe to utilize them. If you follow the instructions on how to use the Fortnite Hack v bucks generator you will receive resources for free.
    We are pleased to announce Fortnite free v-bucks generator for your accounts. We created unlimited v bucks generator for all Fortnite players. This is version 5.2 and contains an option to Get Fortnite V Bucks Generator because there are more and more players on Fortnite servers. You all know how bad everyone needs some v-bucks in-game currency. Feel free to use our v-bucks generator online. You can use it permanently with a maximum of 13.500 V-Bucks daily.

    ReplyDelete
  26. Mit Fortnite kostenlos v bucks Generator erhalten Sie unbegrenzt kostenlose V Bucks. Holen Sie sich die meisten Ressourcen und In App Käufe kostenlos mit dem Fortnite V Bucks Hack Generator. Fügen Sie einfach die auf dieser Site gefundenen Codes in das Fortnite v bucks Generator Werkzeug ein. Der Fortnite V Bucks Hack Generator arbeitet schnell und kann sicher verwendet werden. Wenn Sie die Anweisungen zur Verwendung des Fortnite Hack v bucks Generators befolgen, erhalten Sie kostenlos Ressourcen.
    Wir freuen uns, Ihnen den kostenlosen Fortnite V-Bucks Generator für Ihre Konten vorstellen zu können. Wir haben einen unbegrenzten V Bucks Generator für alle Fortnite Spieler erstellt. Dies ist Version 5.2 und enthält eine Option zum Abrufen des kostenlos v bucks Generator, da sich immer mehr Spieler auf Fortnite Servern befinden. Sie alle wissen, wie dringend jeder ein paar V Bucks in Spielwährung braucht. Nutzen Sie unseren V-Bucks Generator online. Sie können es dauerhaft mit maximal 13.500 V-Bucks täglich verwenden.

    ReplyDelete
  27. I am impressed. I don't think Ive met anyone who knows as much about this subject as you do. You are truly well informed and very intelligent. You wrote something that people could understand and made the subject intriguing for everyone. Really, great blog you have got here. best stereo for toyota rav4

    ReplyDelete
  28. Finally, the latest *New Method* Fortnite Free V Bucks Generator that you have been searching for is here. This is the only hack that has been updated and is currently the only one that actually works. Our Updated Method hack tool is capable of allocating Unlimited Fortnite V-Bucks Cheats, to your account for free and instantly. You do not have to wait for these resources to be added to your account.
    Fortnite Free V Bucks Generator
    Fortnite kostenlos V Bucks Generator
    Fortnite Free V Bucks Generator
    Fortnite Free V Bucks Generator
    Fortnite kostenlos V Bucks Generator

    ReplyDelete
  29. Your work is very good and I appreciate you and hopping for some more informative posts. Thank you for sharing great information to us. Free Fire Mod APK

    ReplyDelete
  30. There are lots of video gaming sites available online, but MMOGAH is really a greatly reputable igaming site. This great site provides not only the easiest delivery support but additionally some special discounts when people order any video gaming currency. To recognise the actual guidance of the online game shop usa , persons can pay a visit to the next weblink https://www.mmogah.com/.

    ReplyDelete
  31. You've chopped up all your brainstorming and assembled some really keen concepts for a storyline and you're ready to go. But amongst all the programming, the character concepts, the dungeons, and the quests - what are truly the most important aspects of your game that will determine whether someone enjoys themself? 토토사이트

    ReplyDelete
  32. This site helps the individuals to buy animal crossing items and a person can obtain the digital currency from this incredible website only via face to face shipping procedure. Avid gamers can experience a secure supply approach animal crossing items for sale on this internet site because it includes qualified team members. The particular team members of this website simply drop the item in the game when you buy items from this great site and you can readily collect all the items.

    ReplyDelete
  33. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article... a1traders

    ReplyDelete
  34. There are several games available on the internet but most folks enjoy playing Star wars the old republic credits: the old republic game. This activity is created via BioWare Austin as well as it's regarded as the massively multiplayer online roleplaying game.

    ReplyDelete
  35. There are many other such games that are played which not only offer enjoyment but roblox gamer also enhance IQ level. Card games such as solitaire, hearts, piquet, bridge, etc are very exciting games that will make you think of different strategies to win the game.

    ReplyDelete
  36. Wide brand exposure can be gained from sponsoring free online games and 안전놀이터 more companies are looking towards game sponsorship and in game product placement as a serious advertising platform.

    ReplyDelete
  37. Today, if you don't have a decent Internet connection in your home, some games are genuinely unplayable, and many others severely hampered.바둑이사이트

    ReplyDelete
  38. This wheel permits me to act in smooth manner like I am driving live out and about. Madden NFL in esports

    ReplyDelete
  39. "먹튀검증 Hello! Nice to meet you. 먹튀검증First of all,안전놀이터 thank you very much for 토토사이트providing us with this variety of information. 꽁머니I think it's information that can be used in many ways.안전놀이터 I can't imagine how much effort you've made to write this. 스포츠중계Thank you for your hard work.꽁머니 I would appreciate it if you would continue to provide us with such information. In addition, https://mtboan.com/ COVID-19 is becoming a problem all over the world these days. 먹튀보안관 I hope there's no harm to you and your family. 먹튀가디언 Thank you https://twiddeo.com/And the information on my site 안전한놀이터.I'm sure you'll admire it.놀이터토토 is awesome. I am sure it will be because I am also surprised by your 꽁머니토토.And if you visit my site, 메이저안전놀이터 I think there will be a lot of information you want 안전놀이터추천Or, if you have the purpose of inquiry, it would be nice to say 먹튀사이트조회Your post 토토사이트 will be of great help 안전놀이터 to the post 먹튀검증 I was preparing. I would like to add your opinion 먹튀요기요 to my post https://mtygy.com/ in the future, so would you please 안전한토토사이트 visit my site once? 사설토토사이트

    "

    ReplyDelete
  40. "토토사이트 Nice to meet you. When was your interest in 안전한토토사이트 when did you start?? I was probably from when I first encountered 안전한놀이터 when I was a child. I am very happy to see someone with the same interests writing such a wonderful article about 먹튀검증커뮤니티.토토사이트분석전문블로그 Your post was really impressive. 먹튀검증의새로운패러다임 It's an article of https://kipu.com.ua/ genius that https://remarka.kz/ I didn't think of. https://subtitles.ee/ I also run a website with 안전놀이터a theme similar to 먹튀검증 that of your article. 토토사이트 I wish I had the talent to 먹튀사이트검증 write with a new perspective like you. 메이저토토사이트 How did you come to 메이저안전놀이터 think of this? 안전한사설토토사이트 I really want to know 공식안전놀이터 if I can. 먹튀검증커뮤니티 Because I want to 먹튀검증토토사이트 write like you. 먹튀검증토토 There are many articles on 사설안전놀이터 my website, but they are not enough compared to you. Could you visit my website if you have time? Please read the posts on my website and leave your thoughts. Then it will be very helpful for me to develop. Thank you. Have a great day.안전놀이터검증및연구센터It's really cool 검증토토사이트. I'd like to know more information. Just like your 안전한먹튀검증 I have 최고의안전놀이터.Nowadays,토토사이트검증업체 the weather is getting colder every day토토안전놀이터, so wear warm clothes and watch out for a cold 확실한먹튀검증Your post 메이저사설토토사이트 will be of great help to the post I was preparing. I would like to add your opinion to my post in the future, so would you please 놀이터추천 visit my site once? 먹튀신고
    "

    ReplyDelete
  41. "토토사이트 Nice to meet you. When was your interest in 안전한토토사이트 when did you start?? I was probably from when I first encountered 안전한놀이터 when I was a child. I am very happy to see someone with the same interests writing such a wonderful article about 먹튀검증커뮤니티.토토사이트분석전문블로그 Your post was really impressive. 먹튀검증의새로운패러다임 It's an article of https://kipu.com.ua/ genius that https://remarka.kz/ I didn't think of. https://subtitles.ee/ I also run a website with 안전놀이터a theme similar to 먹튀검증 that of your article. 토토사이트 I wish I had the talent to 먹튀사이트검증 write with a new perspective like you. 메이저토토사이트 How did you come to 메이저안전놀이터 think of this? 안전한사설토토사이트 I really want to know 공식안전놀이터 if I can. 먹튀검증커뮤니티 Because I want to 먹튀검증토토사이트 write like you. 먹튀검증토토 There are many articles on 사설안전놀이터 my website, but they are not enough compared to you. Could you visit my website if you have time? Please read the posts on my website and leave your thoughts. Then it will be very helpful for me to develop. Thank you. Have a great day.안전놀이터검증및연구센터It's really cool 검증토토사이트. I'd like to know more information. Just like your 안전한먹튀검증 I have 최고의안전놀이터.Nowadays,토토사이트검증업체 the weather is getting colder every day토토안전놀이터, so wear warm clothes and watch out for a cold 확실한먹튀검증Your post 메이저사설토토사이트 will be of great help to the post I was preparing. I would like to add your opinion to my post in the future, so would you please 놀이터추천 visit my site once? 먹튀신고
    "

    ReplyDelete
  42. ""This is the post "" I was looking for "". Your post "" was a great help to "". Thank you for "" providing such good 메이저토토 information. I live in 놀이터토토 a different country from you 먹튀. Even in my country, your post and the information you provide can arouse a lot of sympathy. Also, it will be very helpful. I run a website similar to you in the country I live in. I'm still lacking a lot, but I'm doing my best. If you don't mind, please visit my website and let me know your opinion. It will be very helpful to me.

    ReplyDelete