Results 1 to 9 of 9

Thread: pk3 conflict fixerupper

  1. #1
    Developer Todesengel's Avatar
    Join Date
    Dec 2013
    Location
    St. Louis, Missouri, USA
    Posts
    276

    Default pk3 conflict fixerupper

    For background... James had posted https://www.x-null.net/forums/showth...ll=1#post32626 and I had posted https://www.x-null.net/forums/showth...ll=1#post19355

    I was thinking about this a little today. I think the idea of mods being sent out as pk3's is just not fine-grained enough. It has led to 'dropping in the pk3 works for this guy but not that guy and sorta works for that guy' all due to the conflicts. Using pk3's to replace/modify on a file by file basis is just not ever going to yield great results; it's a sledgehammer when a jewelers hammer is called for.

    In the reborn system, do we have access to affect the .pk3 loading process where the system loads all pk3's and regular directories? If so, I would suggest that mods (could also) be distributed as a package of unified diff/patch files. The new loader process could run the patch utility for each new style 'mod', applying it's diffs to what has already been loaded. Granularity is line by line at worst, and character by character at best.

    I bring this up because the same program I just described could be integrated with what James is thinking above. Thus, his program could not only do the sledgehammer of file by file stuff, but also implement the more 'safe' version I just described. Perhaps automatically if the new mod diff format is .pk4? lol Maybe a command line switch....

    T

  2. #2
    Developer RyBack's Avatar
    Join Date
    Apr 2014
    Location
    In Front of the screen
    Posts
    1,603

    Default

    merging mods into one file would be much easier.

  3. #3
    Administrator James's Avatar
    Join Date
    May 2010
    Location
    on the intraweb
    Posts
    3,180

    Default

    I haven't had a chance to get started on this project yet, but maybe it is worth brain storming some ideas. I def think the idea is a great one and I'd love a "hassle-free" way of loading these archives without needing to do too much extra work.

    Initially I was thinking of using the 7-zip tool to extract the pk3 archive.
    Even after it's extracted it would be a process to scan each character or line and compare it to the other lines to look for differences and then merging everything together. I think if there was a way to load pk3's in some sort of "sandboxed" environment so each pk3 could be activated/enabled on the fly if the admin wants to load a mod or not.

    Lets say some mod is in a.pk3 and another mod with clashing files is in b.pk3. However the admin doesn't really use b.pk3, so he can type a command like disable b.pk3 and only a.pk3 will get loaded. Also the benefit of having a "sandboxed" environment is that it wouldn't be dependent on any of the other files; but I'm not sure if something like this is possible...

  4. #4

    Default

    Quote Originally Posted by James View Post
    I haven't had a chance to get started on this project yet, but maybe it is worth brain storming some ideas. I def think the idea is a great one and I'd love a "hassle-free" way of loading these archives without needing to do too much extra work.

    Initially I was thinking of using the 7-zip tool to extract the pk3 archive.
    Even after it's extracted it would be a process to scan each character or line and compare it to the other lines to look for differences and then merging everything together. I think if there was a way to load pk3's in some sort of "sandboxed" environment so each pk3 could be activated/enabled on the fly if the admin wants to load a mod or not.

    Lets say some mod is in a.pk3 and another mod with clashing files is in b.pk3. However the admin doesn't really use b.pk3, so he can type a command like disable b.pk3 and only a.pk3 will get loaded. Also the benefit of having a "sandboxed" environment is that it wouldn't be dependent on any of the other files; but I'm not sure if something like this is possible...
    I have made a C++ Library to be able to use MOHAA assets/pk3 in external programs, it contains a MOHAA file manager and the library was designed for MOHUE. But I can give you an access to the library and it's API, it's really simple to use and it works like this :


    #include <MOHPC/FileManager.h>
    #include <MOHPC/Asset.h>

    int main(int argc, char *argv[])
    {
    MOHPC::AssetManager AM;

    // Retrieves the file manager associated with the asset manager
    // So we can open a file
    FileManager* FM = AM.GetFileManager();

    // Adds up the main folder from MOHAA
    FM->AddGameDirectory("C:\\Games\\MOHPC\\main");

    // Parses pak0
    FM->AddPakFile("C:\\Games\\MOHPC\\Pak0.pk3");

    // Parses pak1 : Similar files present in pak0 and pak6 will be overriden by pak6
    FM->AddPakFile("C:\\Games\\MOHPC\\Pak6.pk3");

    // Open bar.tik, it will be the bar.tik from Pak6, because it is lastly present in pak6 rather than pak0
    FilePtr File = FM->OpenFile("models/weapons/bar.tik");

    char *Buffer = nullptr;

    // Puts the content of bar.tik into a buffer...
    std::streamsize Size = File->ReadStream(&buffer);

    // No need to close the file, shared_ptr will do it after auto-destruction
    // AssetManager will automatically destroy everything once it begins destruction.
    return 0;
    }

  5. #5
    Developer Todesengel's Avatar
    Join Date
    Dec 2013
    Location
    St. Louis, Missouri, USA
    Posts
    276

    Default

    Quote Originally Posted by RyBack View Post
    merging mods into one file would be much easier.
    I think we have 15 years of conflicts saying that's not the best way to go, IMHO What percentage of the support questions here have a root cause of the server owner thinking that pk3's are plugin/modular oriented when in fact they are not? I suspect that's a pretty high number. The complete syntax definition of moh scripting is available already in a somewhat backus-naur/regex form, included in radiant. A few days with lex/yacc and friends, and you have something that can reliably build a symbol table so that at least conflicts with variable names becomes a simple linear search at load time. At the least - you could generate a warning to the console that a conflict exists, what variable it is, and which pk3 files are involved. Not to mention being able to stop and spit out a syntax/grammar error without having the game start and then crash later.

    I'm just tossing out ideas and brainstorming

  6. #6
    Administrator James's Avatar
    Join Date
    May 2010
    Location
    on the intraweb
    Posts
    3,180

    Default

    T, that's actually a great idea and I think it would be a huge help indeed. Just having a game smart enough to point you to the issue. Console does that already with scripts that are loaded, but you're right I don't think it's smart enough to know if there are 2 or more pk3 files loaded that share resource and can conflict with each other.
    If it were to say lasers.pk3 and bigbob.pk3 have a conflict in scripts\dmprecache.scr on line 36 (unknown character) or something to that degree..

    @Ley0k, that's fantastic!! How long did it take you to make something like this? Does your Library support everything we're trying to do already? Can it parse each file character by character or line by line to determine if there is any conflict in files?
    Or perhaps instead automate the sequence...

    we know that pak0-pak6 are stock, so we'll assume those don't need to be "scanned by the app" so scan all the other pk3 files, compare the files to see if there are any conflicts and then if there are conflicts merge everything together?

    Great job Ley0k! This is very interesting!

  7. #7
    Developer RyBack's Avatar
    Join Date
    Apr 2014
    Location
    In Front of the screen
    Posts
    1,603

    Default

    Quote Originally Posted by Todesengel View Post
    I think we have 15 years of conflicts saying that's not the best way to go, IMHO What percentage of the support questions here have a root cause of the server owner thinking that pk3's are plugin/modular oriented when in fact they are not? I suspect that's a pretty high number. The complete syntax definition of moh scripting is available already in a somewhat backus-naur/regex form, included in radiant. A few days with lex/yacc and friends, and you have something that can reliably build a symbol table so that at least conflicts with variable names becomes a simple linear search at load time. At the least - you could generate a warning to the console that a conflict exists, what variable it is, and which pk3 files are involved. Not to mention being able to stop and spit out a syntax/grammar error without having the game start and then crash later.

    I'm just tossing out ideas and brainstorming
    I thought before of a program that could merge automatically.
    If 2 pk3s has strings.scr we could rename the files and their corresponding calls.
    as for dmprecache or other similar files the'd have to be parsed and inspected for calls or calls etc. then move all calls to a file and call that file from dmprecache.

  8. #8

    Default

    Hey James,

    This library is a sort of remake of the Quake's standard C file manager. Basically it merges up files gradually as you add pk3 files : it is an unified way to open files without needing to take care if they are inside a pk3 or not, and it can load pk3 on the fly (slower than adding them initially however, because files parsing happens once a call to a file function has been made).

    To answer your question, you can open a file, read/seek in a file, and check if there is any conflict (if you use two differents file managers). The file manager is read-only though, meaning that it cannot write file (simply because it was designed to only read assets).
    What I can suggest however, is that you replace those FS_ReadFile/FS_ListFilteredFiles function pointers from the MOH Exports with functions that wrap calls to the new C++ File Manager, by doing so you can make your own pk3 mod loader/unloader, which is more appreciated than resolving conflicts like GIT does.

  9. #9
    Developer Todesengel's Avatar
    Join Date
    Dec 2013
    Location
    St. Louis, Missouri, USA
    Posts
    276

    Default

    Quote Originally Posted by RyBack View Post
    I thought before of a program that could merge automatically.
    If 2 pk3s has strings.scr we could rename the files and their corresponding calls.
    as for dmprecache or other similar files the'd have to be parsed and inspected for calls or calls etc. then move all calls to a file and call that file from dmprecache.
    I agree Ryback. There are also more than just calls to address, see my recent soundfix thread for example... level.bar, level.rifle, etc. Plus, things like diff and patch are already written and well-seasoned. IMHO - I think a good approach for the strings.scr case you mention is to run diff against a stock extraction and supply that diff file in the "mod package". Then at load time you just run patch to apply that diff to the users code. This allows users to have hacked up their files and we still have a pretty good chance of applying the patch - or emit the dreaded "patch did not apply cleanly" message.

    I don't know if youve ever played with diff/patch, but they are pretty good at applying a patch to code that is different than the original/stock one (ie. one that someone has changed themselves). Not perfect, but pretty darn good.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •