Jump to content


Photo

Unravelling the Secrets of "Normality" (1996)

ANALYZE

  • Please log in to reply
117 replies to this topic

#1 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 16 December 2012 - 06:20 PM

a5oI0.gif


I am working out how to rip resources from the game files for the 3D point and click adventure game Normality by Gremlin Entertainment. I strongly suspect there to be many inaccessible/unused/secret things here. I want to look at them.


I will be updating this thread with my progress. I have not got very far yet, many of the files are encoded with a proprietary kind of compression. I will include decompression code once I have reverse engineered it, then I can start working out what's in the raw data


I believe this to be the first project of its kind anywhere on the web... the secrets of Normality will be spilled here, for the first time, live


Also: Analysis of the underlying politics, ideology and intentions of the Normality authors very welcome in this thread
 
 
 
 

iW4PV.gif



#2 dada

dada

    VILLAIN

  • Administrators
  • 17457 posts
  • Locationsuperhell

Posted 16 December 2012 - 07:05 PM

Wowwww
Normality is one of my favorite games ever. I must have played through it about a dozen times. And yeah, there's something MYSTERIOUS about certain things. Like the park at the beginning of the game that you show. I've never been able to get in the normal way, but a while back I recorded some videos that show what happen if you do get in(by modifying the player's coordinates with a debugger).

Normality inaccessible area: the TV station set


Normality inaccessible area: Saul's cell


Normality inaccessible area: the Plush-Rest furniture machine


Normality inaccessible area: the park


If you could find some way to rip the graphics, that would be awesome. I've always sort of had it in the back of my mind to try this but given that it's COMPLEX STUFF that I have no experience with, I've never given it a try.


#3 dada

dada

    VILLAIN

  • Administrators
  • 17457 posts
  • Locationsuperhell

Posted 16 December 2012 - 07:15 PM

Also: Analysis of the underlying politics, ideology and intentions of the Normality authors very welcome in this thread

Check out the comments by BL1TZEN on this vid



#4 Vellfire

Vellfire

    TV people want to leave

  • Salty Members
  • 14593 posts
  • LocationKentucky

Posted 16 December 2012 - 10:56 PM

I was gonna freak out and alert dada about this topic but he already found it and now I feel foolish for thinking that a topic about Normality could show up on this forum and him not be in it IMMEDIATELY.

I love this hobby - stealing your mother's diary
BRRING! BRRING!
Hello!  It's me, Vellfire!  FOLLOW ME ON TWITTER! ... Bye!  CLICK!  @gidgetnomates


#5 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 17 December 2012 - 03:12 PM

R28DY.gif


h'okay, I think I have cracked the decompression algorithm!!

the rest of this post will only be of interest to hardcore bytehackers who want to follow in my filthy footsteps


so, this compression scheme applies to any files that have the extension .MGL and this is some (Lua-ish) pseudocode code to decompress it
repeat
  local b = read_byte()
  if (b == 0) or (b == nil) then
    -- zero byte marks the end of the stream
    break
  elseif b >= 0xE0 then
    local b2 = read_byte()
    local b3 = read_byte()
    local offset = ((b - 0xE0) * 256) + b2 + 3
    local length = b3 + 5
    reuse_bytes(offset, length)
  elseif b >= 0xC0 then
    local b2 = read_byte()
    local offset = ((b % 4) * 256) + b2 + 3
    local length = 4 + math.floor((b - 0xC0) / 4)
    reuse_bytes(offset, length)
  elseif b >= 0x80 then
    local offset = (b - 0x80) + 3
    reuse_bytes(offset, 3)
  elseif b >= 0x70 then
    local reps = (b - 0x70) + 2
    reuse_bytes(2, 2, reps)
  elseif b >= 0x60 then
    local reps = (b - 0x60) + 3
    reuse_bytes(1, 1, reps)
  elseif b >= 0x50 then
    local length = (b - 0x50) + 2
    uint16LE_pattern_sequence(length)
  elseif b >= 0x40 then
    local length = (b - 0x40) + 3
    byte_pattern_sequence(length)
  else
    local length = b
    copy_input_bytes(length)
  end
until (b == 0) or (b == nil)

this is the core loop, it requires the following functions be defined first:
  • read_byte(): Read 1 byte from the input stream, and return its value as a number. If the end of the stream is reached, return nil.
  • copy_input_bytes(length): Take a chunk of bytes from the input stream, and write it direct to the output stream.
  • reuse_bytes(offset, length[, repeats]): Take a chunk of bytes that have previously been written to the output stream, and re-append it to the end. The chunk begins offset bytes backwards from the end of the stream. If length is greater than offset, repeat the chunk like a pattern - for example if reuse_bytes(5, 9) is called and the last five bytes written spell "Hello" in ASCII, "HelloHell" should be written to the output stream. If the optional parameter repeats is given, repeat the chunk that number of times.
  • byte_pattern_sequence(length): Use the last two bytes written to the output stream to continue with a pattern. For example if the last two byte values were 02 03, the sequence would continue 04 05 06 07...
  • uint16LE_pattern_sequence(length): Same as byte_pattern_sequence() but with 16-bit short integers (Little-Endian encoded) instead of bytes. So note the total number of bytes written will be twice the value of length


#6 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 17 December 2012 - 03:21 PM

Normality inaccessible area: the TV station set
Normality inaccessible area: Saul's cell
Normality inaccessible area: the Plush-Rest furniture machine
Normality inaccessible area: the park

 

Check out the comments by BL1TZEN on this vid


Wow this is great stuff!! Thank you, I've been doing searches to find anything Normality-related in preparation for this project but somehow missed your YouTubes completely...

#7 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 20 December 2012 - 10:45 PM

Fot3a.gif


all right this is starting to come together...

first. some more Nerd Shit. this is a Work-In-Progress description doc for the .DAS texture/sprite pack files in the MAPS folder:
 
/*
=======================================
Normality .DAS Texture/Sprite Pack File
=======================================

These files are found in the MAPS folder.
Any .MGL files found here are actually .DAS files that need to be decompressed first.

All multibyte integers are Little-Endian encoded.

(Pseudocode C struct syntax is descriptive only, not intended for actual code use...)
*/

// first in the file
struct File_Header {
  char[6] file_id;        // always "DASP\0\5"
  uint16_t unknown_1;     // 0x8000-0x8600 ?
  uint32_t always_0x28;
  uint32_t addr_palette;  // file address of Palette
  uint32_t unknown_3;
  uint32_t addr_names;    // file address of Name_Table
  uint32_t unknown_4;
  uint32_t unknown_5;
  uint32_t unknown_6;
};

struct Palette_Entry {
  uint8_t red, green, blue; // component values are 0-63 (VGA 6-bit)
};

struct Palette {
  Palette_Entry entries[256];
};

struct Image_Table {
  Image_Record records[?];
};

struct Image_Record {
  uint32_t addr_data; // file address of corresponding Image_Data
  uint32_t unknown;
};

struct Image_Data {
  uint16_t flags; /*
    ANIMATED: ((flags & 0x0100) != 0)
  */
  uint16_t width;
  uint16_t height;
  Animation anim[(flags & 0x0100) ? 1 : 0];
  uint8_t data[width * height];
};

struct Animation {
  uint32_t unknown_1;
  uint16_t length;
  uint16_t unknown_2;
  uint8_t always_0xFFFFFF[3];
  uint8_t speed; /*
    how speed values translate into frame duration (approximate!!!):
      0x02: 70ms
      0x03: 57ms
      0x04: 100ms
      0x06: 128ms
      0x07: 170ms
      0x08: 186ms
      0x0A: 214ms
      0x0E: 270ms
      0x10: 57ms
  */
  uint32_t addr_delta[(length / 4) - 5]; /*
    file address of delta for frame i is
	  the address of the "length" field + addr_delta[i]
      unless addr_delta[i] is zero. then just keep previous frame
    final (non-zero) delta always just restores the original image
      any further zeroes after that frame seem to be meaningless...
  */
  uint32_t unknown_3;
  uint32_t unknown_4;
};

struct Name_Table {
  uint16_t texture_count;
  uint16_t sprite_count;
  Name_Record texture_records[texture_count];
  Name_Record sprite_records[sprite_count];
};

struct Name_Record {
  uint16_t unknown_1;
  
  uint16_t index; /*
    if top bit is set, index is a sprite
    file address is File_Header.addr_textures + (index * 4)
  */
    
  char short_name[?]; // null terminated
  char long_name[?];  // null terminated
};

edit: see this post for info about animation frame delta encoding

here are a few examples of what this can extract so far (edit: out of date!! palette and animation is now understood...)
* starts playing....*


BUILDIN3 "Fluffys" from EUREKA0.DAS
qUw0i.png


SKIPWAL5 "Skip wall5" from DEN.DAS
jj6Er.png


BUSTOP "Bus Stop" from FACTOUT.DAS
bXyIz.png


TUNENT1 "Shoe Shop" from MINTMALL.DAS
sQKkO.png


PILLOW "Pillow!" from EUREKA0.DAS
fbVR0.png


CINDOORS "Cinema Doors" from TRUCKTV.DAS
NWrs7.png


still early days.....

#8 dada

dada

    VILLAIN

  • Administrators
  • 17457 posts
  • Locationsuperhell

Posted 21 December 2012 - 01:45 AM

fantastic, glad you're making progress :)


#9 jamie

jamie

    ruined former youth seeking atonement

  • Salty Members
  • 8247 posts

Posted 21 December 2012 - 02:15 AM

keep going...

#10 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 23 December 2012 - 06:02 PM




Posted Image



good news friends!! i have worked out where the palettes are in the .DAS files


i still don't know how to get the animation frames, and some textures/sprites that i don't think are even animated are still coming out all messed up


but, time for some more ...


Posted ImagePosted Image


Posted Image


Posted Image


Posted Image


Posted Image


Posted Image





#11 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 23 December 2012 - 06:06 PM

bonus: piece on Normality's mocap tech by Violet Berlin for 90s gaming TV show "Bad Influence"


here's the bernard manning placeholder from the early game footage in the clip

Posted Image


don't think he ever appears fully but did get used for one of the CD covers in the music store


Posted Image

#12 Swordfish

Swordfish

    Advanced Member

  • Salty Members
  • 2901 posts

Posted 23 December 2012 - 07:53 PM

I think that what you are doing is super cool, but what's your ultimate end-game? As far as I know your goal is to find secret stuff but I want to know if you got any other ideas.

#13 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 23 December 2012 - 09:11 PM

I think that what you are doing is super cool, but what's your ultimate end-game? As far as I know your goal is to find secret stuff but I want to know if you got any other ideas.


no i don't got any further plans at the moment


but i'd love to see offshoot project(s) if someone wants to take em up, & i'd help where i can


ideas...
 
  • an exporter application
  • modding!! why not propose marriage to your loved one via craftily modified wall textures... or, do we dare to dream, an all-new game on the Normality engine...? don't hold your breath on that tbh... i still have no idea how much of the room objects, events etc. are hardwired into the exe
  • using ultimate papercraft 3D to make a sweet diorama of Kent's apartment, an ideal centerpiece for the modern home
  • faithfully recreating part of the game in Unity3D or w/e


#14 dada

dada

    VILLAIN

  • Administrators
  • 17457 posts
  • Locationsuperhell

Posted 24 December 2012 - 02:51 AM

bonus: piece on Normality's mocap tech by Violet Berlin for 90s gaming TV show "Bad Influence"

Awesome, I knew this existed but I never bothered to look it up. I can't for the life of me figure out what cutscene they're recording for though.


#15 dada

dada

    VILLAIN

  • Administrators
  • 17457 posts
  • Locationsuperhell

Posted 24 December 2012 - 02:52 AM

I would love to make a faithful recreation of Kent's Apartment inside a browser window using maybe something like WebGL.


#16 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 27 December 2012 - 12:15 PM

i've been trying to decode the .GDV cutscene video files... this is actually documented for once http://wiki.multimed...n_Digital_Video


not lookin so good yet though



Posted Image Posted Image
Posted Image Posted Image


there's actually an existing tool out there that can play .GDV files and even export every frame as images, but the reason i'm doin this is I suspect the animated textures/sprites will use same/similar encoding to the video....

#17 dada

dada

    VILLAIN

  • Administrators
  • 17457 posts
  • Locationsuperhell

Posted 27 December 2012 - 12:20 PM

imo that actually looks wonderful in its own right.

Nice to see you're still making progress :)


#18 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 27 December 2012 - 03:20 PM



Posted Image




#19 dada

dada

    VILLAIN

  • Administrators
  • 17457 posts
  • Locationsuperhell

Posted 27 December 2012 - 03:33 PM

whoa cool, is that something you found among the files or is that FANART?


#20 denzquix

denzquix

    Advanced Member

  • Salty Members
  • 625 posts

Posted 27 December 2012 - 03:43 PM

it was in the manual PDF that came with the gog.com installation, i dunno if it was in the original game materials or what





Also tagged with one or more of these keywords: ANALYZE

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users