118 replies to this topic

[jerseys219]

Another game that was made using the Normality engine was Realms of the Haunting. Would it be possible to use the technique described in this thread to extract resources from ROTT as well? Because I'd love to get my hands on some of those assets.

[denzquix]

don't have a copy of roth handy, my guess would be that the file formats would have a lot of similarities but might be slightly different to accommodate new features or get better compression or w/e

[denzquix]

imo that actually looks wonderful in its own right.

agreedo. here's some more recent snaps... think the pixels are all the right colours now, just gotta convince them to go in the right places. closer and closer to deciphering this disasterpiece

[denzquix]

 
yo got the .GDV cutscene video format sorted!! the gif above ^^^^ taken from actual exported vidframes....

 

here is the main codes, for all you chip callahans following along at home, it's a long one:

-- decoding script for Normality .GDV files

if not check_bytes(0x94, 0x19, 0x11, 0x29) then
  error('GDV file header not found')
end

video = {}
audio = {}

read_uint16LE()

video.frame_count       = read_uint16LE()
video.frames_per_second = read_uint16LE()

do
  local packed = read_uint16LE()
  audio.dpcm = (bit.band(packed, 8) ~= 0)
  if bit.band(packed, 4) == 0 then
    audio.bytes_per_sample = 1
  else
    audio.bytes_per_sample = 2
  end
  if bit.band(packed, 2) == 0 then
    audio.channels = 1 -- mono
  else
    audio.channels = 2 -- stereo
  end
  
  audio.present = (bit.band(packed, 1) ~= 0)
end

audio.sample_rate = read_uint16LE()

do
  local packed = read_uint16LE()
  local bpp_enum = bit.band(packed, 7)
  if bpp_enum == 1 then
    video.bits_per_pixel = 8
  elseif bpp_enum == 2 then
    video.bits_per_pixel = 15
  elseif bpp_enum == 3 then
    video.bits_per_pixel = 16
  elseif bpp_enum == 4 then
    video.bits_per_pixel = 24
  end
end

video.max_frame_size = read_uint16LE()

video.present = (video.max_frame_size ~= 0)

read_uint16LE()

video.width = read_uint16LE()
video.height = read_uint16LE()

if video.present then
  init_video(video.width, video.height, video.frames_per_second)
  if video.bits_per_pixel == 8 then
    read_palette()
  else
    error('only 8-bit video is supported')
  end
end

if audio.present then
  audio.chunk_size = math.ceil(
    math.floor(audio.sample_rate / video.frames_per_second)
    * audio.channels
    * audio.bytes_per_sample)
  if audio.dpcm then
    audio.chunk_size = audio.chunk_size / 2
  end
  init_audio(audio.sample_rate, audio.bytes_per_sample, audio.channels)
end

-- bit reader utility
local queue, qsize
function init_bit_reader()
  queue = read_uint32LE()
  qsize = 16
end
local function read_bits(n)
  local retval = bit.band(queue, bit.lshift(1, n) - 1)
  queue = bit.rshift(queue, n)
  qsize = qsize - n
  if qsize <= 0 then
    qsize = qsize + 16
    queue = bit.bor(queue, bit.lshift(read_uint16LE(), qsize))
  end
  return retval
end

function find_color_for_invalid_offset(offset)
  local result = bit.band(0xFE, bit.rshift(bit.bnot(offset), 3))
  local lastbit = bit.band(0xF, offset)
  if lastbit == 0 then
    result = bit.band(0xFF, result + 2)
  elseif lastbit <= 8 then
    result = bit.band(0xFF, result + 1)
  end
  return result
end

-- frame decoders
frame_decoders = {}

frame_decoders[0] = function(frame)
  read_palette()
end

frame_decoders[1] = function(frame)
  read_palette()
  video_clear(0)
end

frame_decoders[3] = function(frame)
  -- do nothing!
end

local decoder_6_subdecoders = {}

decoder_6_subdecoders[0] = function()
  if read_bits(1) == 0 then
    write_pixel(read_uint8())
    return
  end
  local length = 2
  local count = 0
  local step
  repeat
    count = count + 1
    step = read_bits(count)
    length = length + step
  until step ~= bit.lshift(1, count) - 1
  for i = 1, length do
    write_pixel(read_uint8())
  end
end

decoder_6_subdecoders[1] = function()
  if read_bits(1) == 0 then
    video_advance(read_bits(4) + 2)
    return
  end
  local b = read_uint8()
  if bit.band(b, 0x80) == 0 then
    video_advance(b + 18)
    return
  end
  local b2 = read_uint8()
  video_advance(bit.bor(bit.lshift(bit.band(b, 0x7F), 8), b2) + 146)
end

decoder_6_subdecoders[2] = function()
  local subTag = read_bits(2)
  if subTag == 3 then
    local b = read_uint8()
    local length = 2
    if bit.band(b, 0x80) == 0x80 then
      length = 3
    end
    local offset = bit.band(b, 0x7F)
    if offset == 0 then
      if get_video_pos() == 0 then
        repeat_pixel(0xFF, length)
      else
        repeat_pixel(read_pixel(-1), length)
      end
      return
    end
    offset = offset + 1
    if offset > get_video_pos() then
      local set_pix = find_color_for_invalid_offset(offset - get_video_pos())
      repeat_pixel(set_pix, length)
      return
    end
    copy_pixels(-offset, length)
    return
  end
  local next_4 = read_bits(4)
  local offset = bit.bor(bit.lshift(next_4, 8), read_uint8())
  if subTag == 0 and offset == 0xFFF then
    return 'stop' -- end of stream
  end
  if subTag == 0 and offset > 0xF80 then
    local length
    length, offset = bit.band(offset, 0xF) + 2, bit.band(bit.rshift(offset, 4), 7)
    local px1 = read_pixel(-(offset + 1))
    local px2 = read_pixel(-offset)
    for i = 1, length do
      write_pixel(px1)
      write_pixel(px2)
    end
    return
  end
  local length = subTag + 3
  if offset == 0xFFF then
    if get_video_pos() == 0 then
      repeat_pixel(0xFF, length)
    else
      repeat_pixel(read_pixel(-1), length)
    end
    return
  end
  offset = 4096 - offset
  if offset > get_video_pos() then
    local set_pix = find_color_for_invalid_offset(offset - get_video_pos())
    repeat_pixel(set_pix, length)
    return
  end
  copy_pixels(-offset, length)
end

decoder_6_subdecoders[3] = function()
  local first_byte = read_uint8()
  local length = bit.rshift(first_byte, 4)
  if length == 15 then
    length = length + read_uint8()
  end
  length = length + 6
  local offset = bit.bor(bit.lshift(bit.band(first_byte, 0xF), 8), read_uint8())
  if offset == 0xFFF then
    if get_video_pos() == 0 then
      repeat_pixel(0xFF, length)
    else
      repeat_pixel(read_pixel(-1), length)
    end
    return
  end
  offset = 4096 - offset
  if offset > get_video_pos() then
    local set_pix = find_color_for_invalid_offset(offset-get_video_pos())
    repeat_pixel(set_pix, length)
    return
  end
  copy_pixels(-offset, length)
end

frame_decoders[6] = function(frame)
  set_video_pos(frame.offset)
  init_bit_reader()
  local subdecoder
  repeat
    subdecoder = decoder_6_subdecoders[read_bits(2)]
  until subdecoder() == 'stop'
end

decoder_8_subdecoders = {}

decoder_8_subdecoders[0] = decoder_6_subdecoders[0]

decoder_8_subdecoders[1] = decoder_6_subdecoders[1]

decoder_8_subdecoders[2] = decoder_6_subdecoders[2]

decoder_8_subdecoders[3] = function()
  local first_byte = read_uint8()
  if bit.band(first_byte, 0xC0) == 0xC0 then
    local top_4 = read_bits(4)
    local next_byte = read_uint8()
    length = bit.band(first_byte, 0x3F) + 8
    offset = bit.bor(bit.lshift(top_4, 8), next_byte)
    copy_pixels(offset + 1, length)
    return
  end
  local length, offset
  if bit.band(first_byte, 0x80) == 0 then
    local bits_6_to_4 = bit.rshift(first_byte, 4)
    local bits_3_to_0 = bit.band(first_byte, 0xF)
    local next_byte = read_uint8()
    length = bits_6_to_4 + 6
    offset = bit.bor(bit.lshift(bits_3_to_0, 8), next_byte)
  else
    -- read bits BEFORE read byte!
    local top_4 = read_bits(4)
    local next_byte = read_uint8()
    length = 14 + bit.band(first_byte, 0x3F)
    offset = bit.bor(bit.lshift(top_4, 8), next_byte)
  end
  if offset == 0xFFF then
    if get_video_pos() == 0 then
      repeat_pixel(0xFF, length)
    else
      repeat_pixel(read_pixel(-1), length)
    end
    return
  end
  offset = 4096 - offset
  if offset > get_video_pos() then
    local set_pix = find_color_for_invalid_offset(offset-get_video_pos())
    repeat_pixel(set_pix, length)
    return
  end
  copy_pixels(-offset, length)
end

frame_decoders[8] = function(frame)
  set_video_pos(frame.offset)
  init_bit_reader()
  local subdecoder
  repeat
    subdecoder = decoder_8_subdecoders[read_bits(2)]
  until subdecoder() == 'stop'
end

for i = 1, video.frame_count do
  if audio.present then
    copy_audio_data(audio.chunk_size)
  end
  
  if video.present then
    if not check_bytes(0x05, 0x13) then
      error('header for video frame #' .. i .. ' not found')
    end
    local frame = {}
    frame.size = read_uint16LE()
    do
      local packed = read_uint32LE()
      frame.encoding = bit.band(packed, 15)
      frame.offset = bit.rshift(packed, 8)
      frame.half_resolution_mode = bit.band(packed, 32) == 32
      frame.quarter_resolution_mode = bit.band(packed, 16) == 16
      frame.show = bit.band(packed, 128) == 0
    end
    frame.start = get_input_stream_pos()
    local decode = frame_decoders[frame.encoding]
    if not decode then
      error('frame #' .. i .. ' has unsupported encoding type: ' .. frame.encoding)
    end
    if not frame.show then
      -- send the previous frame through again
      output_video_frame()
    end
    if frame.quarter_resolution_mode then
      set_resolution_mode('quarter')
    elseif frame.half_resolution_mode then
      set_resolution_mode('half')
    else
      set_resolution_mode('full')
    end
    decode(frame)
    if frame.show then
      output_video_frame()
    end
    set_input_stream_pos(frame.start + frame.size)
  end
  
end

finish_audio()

 
...which needs the following functions to be available:

input data stream

• read_uint8(): read unsigned 8-bit integer. return as a number value
• read_uint16LE(): read unsigned 16-bit integer, little endian encoded. return as a number value
• read_uint32LE(): read unsigned 32-bit integer, little endian encoded. return as a number value
• check_bytes(...): for each parameter, read a byte from the input stream. return true if every parameter is equal to the corresponding byte, otherwise return false
• get_input_stream_pos(): return the current position of the input stream
• set_input_stream_pos(pos): set the current position of the input stream

video output

• init_video(width, height, frames_per_second): allocate a block of pixels width * height for video read/write operations to work on
• read_palette(): read 256 x 3-byte RGB triplets from the input stream and set them as the current palette. each RGB component is a 12-bit value (0-63)
• video_clear(v): clear the video frame, so that every pixel has the given value v
• set_resolution_mode(mode): if mode is 'half', all pixel read/write operations must operate as if the vertical resolution is halved, i.e. pixels are twice as tall. if mode is 'quarter', all pixel operations must operate as if both vertical and horizontal resolution is halved, i.e. pixels are twice as wide and twice as tall. if mode is 'full' (the default), restore normal operation.
• set_video_pos(absolute_pos): move the pixel-write position to absolute_pos. The position value is related to x/y co-ordinates as follows: (y * video.width) + x == absolute_pos
• get_video_pos(): return the current pixel-write position
• video_advance(relative_offset): move the pixel-write position ahead from its current position by relative_offset pixels.
• write_pixel(v): write a pixel at the current pixel-write position then advance the position by 1
• read_pixel(offset): read back the pixel value at offset bytes relative to the current position. do not change the current position
• repeat_pixel(v, n): write the same pixel repeatedly n times, advance the current position by n
• copy_pixels(offset, length): take a chunk of pixels from offset relative to the current position
• output_video_frame(): signal that the current video frame is complete, and the current pixel/palette data should be sent to screen/file

audio output

• init_audio(sample_rate, bytes_per_sample, num_channels): set up audio output
• copy_audio_data(length): take a chunk of data from the input stream with the given length (in bytes) and send it to audio output
• finish_audio(): close/clean up the audio output system if necessary

 

phew.....

[dada]

nice job

[ATARI]

this is like my favorite art ever.

[denzquix]

unused (?) cutscene: (EDIT: not unused at all!! just an alternate puzzle solution...)

also thanks to http://cd.textfiles.com for the march 1996 edition of "CD Zone" which includes a playable demo, I haven't managed to run it but I can extract stuff like this promo video:

[dada]

unused (?) cutscene:
instead of the sharkpoon kent uses some kinda green thing... what is that thing!!

That's another way to smash the washing machine, you get some frozen *FOOD* from the MINT mall and wrap it in the towel. Apparently you can also smash it using the scissors but in that case there's no video. I guess maybe they thought the food/towel route was too difficult?

Oh, could you put up the full version of that promo video?

[denzquix]

That's another way to smash the washing machine, you get some frozen *FOOD* from the MINT mall and wrap it in the towel.

ohhhh oops i only just found out you can even take the towel

maybe i should get my hands on a copy of this to avoid further embarrassment....

Oh, could you put up the full version of that promo video?

sure thing

[dada]

sure thing

haha, cool. Never seen this.

So, what do you think is next? You've got textures, videos, anything else planned?

[denzquix]

well, the point of getting videos out was to see if it would help with the animated textures (& sprites)... i haven't got back to that yet. i wanna get a crisp version of that bouncing space hopper at least


i did manage to get the corner of Kent's TV to bulge out across the room by messing with some data in MAPS/EUREKA0.RAW so that is confirmed as the geometry file format I guess...

v. useful for watching normtext in bed....

[dada]

yeah I once messed around with some of the filenames, renaming 0.raw to 1.raw etc. and that caused things to be completely messed up. there don't seem to be that many sanity checks in place.

[Mateui]

I've never heard of this but now I'm intrigued. Need to track this game down and experience it.

[dada]

they're selling it on gog.com these days, in case you want an easy way to get it.
http://www.gog.com/gamecard/normality only $6 too.

[dada]

Where R My Updates.......

[denzquix]

oh er *scrabbles around*

ok here's something. i've decoded the game text files

contents of LANG.DAT: http://pastebin.com/0usyhpxS

contents of ENGLISH.DAT (from the demo): http://pastebin.com/yrutiGxh

these files are real simple. the first part is a list of file addresses (little-endian uint32s), followed by the strings (null-terminated), up to the end of the file. the file addresses point to the start of the strings except in some cases they are just zero, these lines appear as '-' in the contents files above. i just used the first address found in the file as the end of the address list.

the demo one's got text from the whole game, here are some things that got added or removed between versions (not all of em):

---

"\"Kent Knutson: Offences include infringement of the Joyous Abstention Laws, harassing a Norm Trooper in the pursuit of his duty, sabotaging a node of Leader Paul's Mood Magnet, terrorism, and others! Evidence for these crimes has been processed, and has been found to be conclusive. Arrest on sight!\""

"I'm not a trashman! I'm a scientist!"

"What...?! Sparks, like the ones in that room where I met Paul, nasally congested fool that he is."

"It's an old T-shirt with \"I AM FAT AND PROUD\" on the front and \"HONEST, REALLY\" on the back"

"I stole it from a hotel down the road. It has great insulation properties." "This towel's good for keeping things hot or cool. Terrific insulation properties."

"Bright stuff! There isn't enough naturally-occuring yellow in this city for my liking." "This has gone all manky. I like yellow, but this is no good."

"Not with my bare hands! They're not God's most hygenic creatures!""I need something small to put him in first."

"Want to meet some people who share the same interests? Then don't just sit being bored at home, get down to the third dumpster in the dingy alley behind the Plush-Rest Furniture Factory." "Frustrated? Want to meet some like-minded individuals? Want to do something about the state of the city? Get a job and snoop around Plush-Rest. You'll find a group in one of the dumpsters around the back."

"OY! Don't steal from me! I'll lose my job, and that's all I have in the world... much as I hate it."

"All in a days work, Citizen. Here, have a badge."
"Get away from that! Here, have a badge."
"Sure I'm sure. Have a badge, and don't distract me again, Citizen."

"It's a fire extinguisher devoid of its yellow gunge without any foam in it."

"I don't think the guy would let me in. He looks kinda pissed cheesed off!"

"My pride and joy. I'm chuffed glad that it worked."

"The prisoner ledger. Let's see... Mr. Ab Normal, doing bird time for impersonating Paul Nystalux. Thirty years ago! Aha, that MUST be Saul!"

"Wait a minute... haven't I wandered into the wrong movie game?!"

"I suppose I could carry one can, just in case of emergency... for when there's nothing else to eat... and I've eaten all my clothing... and both legs."
"Lovely! Hmmm... 'THE ONE AND ONLY *FOOD*. GUARANTEED NO WASTE, MORE WAIST, WHAT TASTE? INGREDIENTS: MEAT AND VEGETATION.' Why am I suddenly not hungry?"

"I haven't the heart to set it again."
"Yuk! Not with that dead rat hanging off!"
"They fall for it every time!"

"I'll just wait for a bus..."

"Just me? Aren't you guys gonna help? I'm no one-man army. Plus, I'm a wanted man already. I'm too young to be tortured!"

[dada]

"I suppose I could carry one can, just in case of emergency... for when there's nothing else to eat... and I've eaten all my clothing... and both legs."

If you use the game's change level cheat (to go to the Plush Rest Factory I think), it will actually give you the can of food item. There's no way to get it in the actual game. I don't think it does anything.
edit: by the way, is it possible to make a dump of the LANG.DAT file that only contains actual things Kent or other people say? as in, without all thost "JUNK", "sharkpoon" etc lines?

[denzquix]

edit: by the way, is it possible to make a dump of the LANG.DAT file that only contains actual things Kent or other people say? as in, without all thost "JUNK", "sharkpoon" etc lines? 

no i have no information about how text is used yet it doesn't seem to be part of the LANG file itself it's just a big list of lines

sorry


but i think it's time for a... sewer rat secret

ok so this guy is known only as `King Rat` when you mouse-over him
 

but thanks to sprite metadata I can confirm that he does have a name and it is...

...of course...

..."Sphincter"
  

proof. sadly the Ninja Turds do not seem to have individual names

i have another sewer rat secret but i will leave it for another time. it is an even bigger bombshell than this one. all exclusively right here on salt world dot net

[dada]

haha 'sphinctr'.
as I said in PM I'll try to obtain a US version of the game for Further Examination.

[mkkmypet]

huh, i have never played Normality, but now i really want to!! :o all of this stuff looks super rad, man. nice work on this, denzquix! :D