Extra bytes being sent/received

Tar.Aldarion

I am using a terminal emulator library to create a terminal and then I use it to send the data entered over serial to a serial device. The library can be seen here: https://github.com/jackpal/Android-Terminal-Emulator/tree/master/libraries/emulatorview


When I enter data into the terminal a strange series of characters is being sent/recieved.

Screenshot of what appears in the terminal when i write "test":
http://i.imgur.com/Pxn2LhA.png


And the log showing the strings sent and the data received.
http://i.imgur.com/x79aPzv.png

Is there any way to fix this without parsing out characters, this might prove annoying if I actually need to type those characters!


When I type in the terminal this is run:

    public void write(byte[] bytes, int offset, int count) {

             String str = new String(bytes);
             GraphicsTerminalActivity.sendOverSerial(bytes);
             //if I uncomment this line the correct character is written to the terminal
             //even though I see the same weird  question mark diamonds in the log
             //because of how my app is laid out this cannot be used, I must call sendOverSerial
             // appendToEmulator(bytes, 0, bytes.length);
             return;
    	 }

This is what I call to send data. sendData(Byte[] data) is a library method.

    public static void sendOverSerial(byte[] data) {
    		
            if(mSelectedAdapter !=null && data !=null)
               {
    		        mSelectedAdapter.sendData(data);
               }

Once data is sent the reply is received here:

    public void onDataReceived(int id, byte[] data) {
    		
    		dataReceived = new String(data);
            dataReceivedByte = data;
    		statusBool = true;
    		((MyBAIsWrapper) bis).renew(data);
    
    		runOnUiThread(new Runnable(){
    
    			@Override
    			public void run() {
    				//As you can see the same command that correctly wrote the
                    //character in the other class is called again, however now it writes the extra bytes to screen.
    				mSession.appendToEmulator(dataReceivedByte, 0, dataReceivedByte.length);
    				
    			}});
    			
    		viewHandler.post(updateView);
    	}

So it works well if I don't send if over serial, even though I can see weird bytes in the logs too, after I send them over serial and they return however they actually get written to the terminal?

Anyway to solve this without just parsing out occurrences of ~0?

CreepingDeath

Looks like a character encoding issue.

Characters are 4 bytes long in Java, and I see 3 extra 0~ in your output per character, which is no coincidence.

So I think the serial comms is probably right, but I suspect you're sending bytes instead of full characters to the screen.

In the onDataReceived() method you do the right thing... ie. " String str = new String(bytes);" to convert the bytes to a string.

* Although it may be more reliable to fix the character set.
You might have an issue with character set translation when you use the default one implicitly in "new String(bytes);"
If the receiving terminal had a different default character encoding scheme, it would mess things up.

But I think the issue is with the "mSession.appendToEmulator(dataReceivedByte, 0, dataReceivedByte.length);" call.

You have correctly translated the bytes to a string above, but then you send raw bytes to the "mSession" ?

Does the mSession.appendToEmulator() method just dump out the bytes to the screen?
If so, it should be taking 4 bytes per character and displaying it.

regards,
CD

Tar.Aldarion

Yeah i thought that was no coincidence all right, that the extra three bytes are 0, just not sure how to handle it. Also why the ~?
As for String str = new String(bytes); I have been messing around with different ways to write to the screen, I had another built in library method that wrote strings but it wrote even weirder stuff, [j before every character and things like that. Also the design of my program now I can't use that method here without causing an infinite loop calling the same method over and over.
So I'm stuck with append, I just need to append the right thing to the screen. I'm not sure why it doesn't write the extra bytes to the screen until after being set over serial and back. I can see that they are there beforehand in the form of little diamonds with ?'s inside them, in the log.

Append does just write bytes to the screen, definition:

Write something directly to the terminal emulator input, bypassing the emulation client, the session's InputStream, and any processing being done by processInput.

I'll try some character sets now this morning.

PrzemoF

OP, that might be interesting for you: http://www.joelonsoftware.com/articles/Unicode.html

Tar.Aldarion

Thanks< I can see why I receive thoise odd traingle things now, it not knowing how to display the data. But it still doesn't work if I make everything UTF-8?

So say I press t on the terminal, I can see t��� sent to the write method, so i tried to change that to UTF-8 and send it around but that doesn't work, is it because it doesn't know waht � is? I tried this, and did this in the other methods too, changing to a UTF-8 string and then back to a byte array:

 public void write(byte[] bytes, int offset, int count) {

		
		 String str;
		try {
			str = new String(bytes, "UTF-8");
			  Log.d(TAG, "data received in write: " +str );
			  
              GraphicsTerminalActivity.sendOverSerial(str.getBytes("UTF-8"));
		} catch (UnsupportedEncodingException e) {
			Log.d(TAG, "exception" );
			e.printStackTrace();
		} 		
		 return;
	 }

PrzemoF

I deleted my previous post - I just found info that android always use UTF-8 as default, so my advice shouldn't make any difference.
getBytes() should work the same as getBytes("UTF-8")

Tar.Aldarion

No, I see no difference at all anyway. I tried changing to UTF-32 and nothign worked at all, every single character was unknown

CreepingDeath

Can you post the code which calls "public void write(byte[] bytes, int offset, int count) "

In particular how you get from the string "test" to the array of bytes sent in that first parameter.

Tar.Aldarion

When I hit a key on the terminal my write method is automatically called, I have no idea what code in the library does this. I'll have a look though it to try and see now. https://github.com/jackpal/Android-Terminal-Emulator/tree/master/libraries/emulatorview The logical thing to do all right!

Tar.Aldarion

I see this in EmulatorView.java I create an EmulatorView, it's the terminal view. it mentions the diamonds: https://github.com/jackpal/Android-Terminal-Emulator/blob/master/libraries/emulatorview/src/jackpal/androidterm/emulatorview/EmulatorView.java

private void sendText(CharSequence text) {
                int n = text.length();
                char c;
                try {
                    for(int i = 0; i < n; i++) {
                        c = text.charAt(i);
                        if (Character.isHighSurrogate(c)) {
                            int codePoint;
                            if (++i < n) {
                                codePoint = Character.toCodePoint(c, text.charAt(i));
                            } else {
                                // Unicode Replacement Glyph, aka white question mark in black diamond.
                                codePoint = '\ufffd';
                            }
                            mapAndSend(codePoint);
                        } else {
                            mapAndSend(c);
                        }
                    }
                } catch (IOException e) {
                    Log.e(TAG, "error writing ", e);
                }
            }

Tow

Speed / Port Parameter mismatch? Are both devices serial ports set the same. eg 9600 8N1 etc...

Tar.Aldarion

I think the port speeds are fine, if I send a string I get the correct string back, the issue is I am sending odd bytes over serial in the first place. I think something got to do with this EmulatorView.java class.

Tar.Aldarion

So I think what is happening is that the unicode replacement character is appended 3 times to the end of each character. ��� These are sent over serial and the serial device doesnt know what it is receiving, it sends back ~0. Somewhere in the library has it set up like this for some reason. I need to not send the � characters over serial, how would i even refer to them in java to parse them out? I can't say if (!str.contains("�") I take it. I suppose I could specify to only sent standard letters and numbers. I can use a regex but that's a last resort kinda fix.

Tow

Serial RS232 is normally 8N1, ie 8 bits ascii. I know nothing about Java (it did not exist back in the day), but this java library has two read/write functions, one is for normal 8 bit ascii bytes and the other is for a java UTF8 string of 2 size bytes first and a unicode (2 byte) char, which would appear to be the 4 bytes you are seeing on the DTE, as it expects ascii.

Tar.Aldarion

I'll have a read of that thanks, and try and reconcile it with the class in the library that I think is causing it:
https://github.com/jackpal/Android-Terminal-Emulator/tree/master/libraries/emulatorview

Current theory is somewhere, raw 32-bit Unicode data was converted to bytes without being encoded. From that point forward, either sending it as-is or re-encoding it as UTF-8 would produce the effect I'm seeing.


Relevant section of class in library:

private void sendText(CharSequence text) {
                int n = text.length();
                char c;
                try {
                    for(int i = 0; i < n; i++) {
                        c = text.charAt(i);
                        if (Character.isHighSurrogate(c)) {
                            int codePoint;
                            if (++i < n) {
                                codePoint = Character.toCodePoint(c, text.charAt(i));
                            } else {
                                // Unicode Replacement Glyph, aka white question mark in black diamond.
                                codePoint = '\ufffd';
                            }
                            mapAndSend(codePoint);
                        } else {
                            mapAndSend(c);
                        }
                    }
                } catch (IOException e) {
                    Log.e(TAG, "error writing ", e);
                }
            }

            private void mapAndSend(int c) throws IOException {
                int result = mKeyListener.mapControlChar(c);
                if (result < TermKeyListener.KEYCODE_OFFSET) {
                    mTermSession.write(result);
                } else {
                    mKeyListener.handleKeyCode(result - TermKeyListener.KEYCODE_OFFSET, getKeypadApplicationMode());
                }
                clearSpecialKeyStatus();
            }