support binary where possible?

[dominictarr]

One great thing about websockets is that it supports binary.
so do some of the other fallbacks, but not all.

What if you allowed binary on those that do,
perhaps wrapping messages in say, the redis protocol?

(as it's binary friendly and there are likely to be implementations in most languages)

Here is one is javascript:
https://github.com/mranney/node_redis/blob/master/lib/parser/javascript.js

Where binary is not supported, using base64 encoding.
That expands the messages, but currently, if you wish to do binary over sockjs, you have to use base64 anyway.

[dominictarr]

Would you be open to a pull request for such a feature if it was backwards compatible with current sockjs?

[majek]

The problem isn't that much in a protocol, it's about making sockjs-client behave in old browsers that don't support typed arrays. Additionally making it work over iframe transports. Another question is: should we experiment with binaries over xhr? (xhr can send binary data, at least in theory).

[majek]

My problem with binaries is the use case - can you tell me why you need to send binary data to a browser? Or do you need to get it from the browser?

[dominictarr]

Okay, thanks.

We need two-way binary between server and browser.

We have been working on this project https://github.com/rvagg/node-levelup
Building databases in javascript, on top of leveldb
we also have polyfils so that it works in the browser and a client/rpc protocol multilevel

@maxogden is using this a voxel.js thing blockplot,
and other stuff too, like encrypted streams, such as secure-peer

I know that max is using straight websockets (which support binary) we are generally not too worried about
old browsers, but of course, sockjs isn't just about old browsers, it's also about proxies and so on that might not support websockets. In any case, I've always found sockjs to work very well, except binary support is it's biggest weakness.

[rstoyanchev]

BinaryJS lists some use case under "What can you do with this?". Almost all imply some way of streaming by splitting up whatever is being sent into smaller parts and re-assembling it on the other end. BinaryJS does this via BinaryPack "serialization" although from a superficial check it looks more like a minimal protocol on top (sort of like the mux extension to SockJS). The future plans for BinaryJS btw lists fallback options (fwiw xhr is listed there).

Given the difficulty of supporting binary data on all browsers, perhaps it would be okay to go as far as possible only and not support all. That's still useful IMO. If you have IE 6/7, then you probably should get a message that your browser is too old to support a feature that requires binary streaming.

[dominictarr]

Well, getting what binary js does is straightforward.
you can also just use something like https://github.com/dominictarr/msgpack-stream
to encode js objects (including node Buffers/TypedArrays) into a websocket stream.
this has the significant advantage that the protocol is decoupled from the networking code,
so, it's possible to stream your binary encoded js data over http or stdio or whatever.

It's even possible to stream over sockjs, except that you have to encode it as base64,
which is a shame in the case that the underlying protocol actually supports binary.

Also, to really make this optimal I understand that it's best to avoid doing a memory copy, and
just writing directly to the socket. now, this isn't allways going to be possible,
but it is desirable that the best case is as good as possible.

[JokerQyou]

Well, seems like a pretty old issue? Sorry to bother you, then.
I'm here to answer @majek 's question.

My problem with binaries is the use case - can you tell me why you need to send binary data to a browser? Or do you need to get it from the browser?

In my case, I'm trying to use MQTT protocol to communicate a MQTT broker (message delivery server), and MQTT is a binary protocol. Since Paho MQTT library uses WebSocket as transportation layer, it may not support some old browser like IE8 (which is still a common version used here in China). So I'm trying to wrap Paho's WebSocket in Socket.io or Sockjs or whatever library that is compatible with some old browsers.
Hope this helps. As to some projects, the ability to transfer binary data across network is as important as any other regular data types, because binary data itself is a common data type.

[brycekahle]

It isn't that we don't want binary support, but it is a significant amount of work. Without a lot of demand, priorities are going to go elsewhere.

[JokerQyou]

@brycekahle Well I understand that. I was just describing one of the many usages of binary data type.
Thx for your reply. :)

[marcelocantos]

Another use case in response to majek's question: sending protocol buffers between servers, modern browsers, and native mobile apps. I want to use SockJS for its solution to the fallback problem, and don't care about older browsers. (Why protobufs? Compact binary format; strongly typed in C++ and Java; elegant schema evolution strategy.)

I can easily base64 everything in and out, but it's a waste of bandwidth if the transport happens to be websocket. This should be decided intelligently and transparently by the transport layer.

Wrt the significant amount of work, dominictarr never got an answer to whether you'd be open to a pull request. Would you?

[brycekahle]

Definitely always open to PRs. They need to maintain support for older browsers, but only supporting binary where it is available is OK, as long as it handles it gracefully.

[dominictarr]

I was thinking that in binary mode, you'd fallback to a base64 encoding when you are over a transport that can't do binary, and then use binary when you are over websockets (etc)

[m11y]

Any updates on this?