Skip to content

October 17, 2014

85

VMware PXE Limitations

Update 12/12/2014: While the information below is valid that VMware does not appear to support RamDiskTFTPBlockSize above 1432 due to no support to deal with IP Fragmentation.  However, it has been found that it is much better to adjust RamDiskTFTPWindowSize instead of RamDiskTFTPBlockSize to speed up TFTP (reduces amount of ACKs required and does not cause IP Fragmentation).  VMware and other vendors all appear to handle this scenario perfectly. Modifying the WindowSize is easy to do from most PXE providers, including WDS (modify BCD).  There is an issue changing this setting in Configuration Manager due to dynamic BCD that ConfigMgr manages itself with no native way to modify the setting  (see the comments section for how to workaround this issue – thanks Patrick!).  More info throughout the comments section.

Update 3/18/2016: Microsoft has released an update (currently in preview) that will now allow modifying both the WindowSize and the BlockSize without a hacked DLL. Info here: https://technet.microsoft.com/en-us/library/mt595861.aspx#BKMK_RC1603. This update is in 1603 Technical Preview and should also be apart of the next Current Branch release of Configuration Manager.

Update 7/22/2016: Microsoft has released a new Current Branch of SCCM, version 1606. I have confirmed that Current Branch version 1606 contains these modifications that were in version 1603 Preview. I have also tested the WindowSize/BlockSize registry keys and they are working great.
Example:
Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP
Type: DWORD
Name: RamDiskTFTPWindowSize
Value: 6
Note: After making changes, make sure to restart the Windows Deployment Service.

We recently ran into some issues, where on various hardware settings, PXE (on physical hardware from various vendors) was very slow (10 minutes to copy 150 MB).  After doing some research we found this issue could be easily rectified by changing the RamDiskTFTPBlockSize to a large value (such as 16384).  We use SCCM 2012 R2 for our PXE server, but this setting can be changed directly on WDS, or SCCM 2007 as well.  I also believe the same change can be made on other third party PXE providers as well, I just haven’t tested those.  Here is a good blog on how to change your RamDiskTFTPBlockSize for various environments: http://windowsdeployments.net/how-to-speed-up-pxe-boot-in-wds-and-sccm/

We found as soon as we made the RamDiskTFTPBlockSize change, PXE was lighting fast everywhere.  Examples from our testing are: Microsoft Hyper-V VM, Dell, HP, Wyse, 10ZiG.  Everything was great, except for VMware, where PXE boot now fails.

We tried using the following settings for the block size: 2048, 4096, 8192, 16384 and they all fail on VMware with error code 0xC0000001.  These values worked fine on all other systems.  When using the E1000 NIC, the info on the error states: “A required device isn’t connected or can’t be accessed”.  When using VMXNet3, the info on the error states “File: \Windows\System32\boot\winload.efi; Info: The application or operating system couldn’t be loaded because a required file is missing or contains error”.  The failures occur whether the firmware is BIOS of UEFI (error message just slightly deviates).  Based on the errors, it was clear that VMware was failing the TFTP transfer and leaving corrupt data.

To try to find the source of the problem, we ran packet captures in VMware to see where the breakdown was happening.  We also sent these off to VMware Support for analysis (getting VMware support involved was actually a very painful process; which took over a month to get an answer back, despite calling and emailing support frequently with no response).  Once VMware finally got back to us the answer unfortunately was not very positive.  VMware states: 1) IP reassembly is not implemented in PXE code and 2) ESXi PXE does not support jumbo frames.

This basically means that VMware cannot support RamDiskTFTPBlockSize being any bigger than 1432.  This is a pretty big limitation since this affects our ability to reasonably support our physical machines and VMware machines at the same time.  It also seems bad since VMware is the only vendor I can find that has this problem.  It wouldn’t be a huge problem if at least supported 8192, but not even 2048 works.

If anyone else has this problem and has found a good workaround, please leave notes in the comments below.  Hopefully this information will save some troubleshooting time.

 

 

Read more from VMware
85 Comments Post a comment
  1. Nov 11 2014

    I’m experiencing a very similar and probably related problem.
    We are using Windows Server 2012 R2 with vmxnet3, SCCM 2012 R2 with CU3 and booting from PXE on site servers.

    Most locations are taking 10 minutes but some are taking up to 40 minutes booting from a local server.

    I also have tried to set the RamDiskTFTPBlockSize as well as WindowSize to all possible values.

    Differently from you I can notice the difference when I set those values but for worse not for better!

    Looking for possible solutions.

    Cheers

    Reply
    • Brent
      Nov 11 2014

      Interesting. Are you ever able to successfully when PXE booting in VMware with RamDiskTFTPBlockSize larger than 1432; or does it always come back failed (corrupt?) – in addition to being worse in time?

      Reply
  2. Nov 11 2014

    Yes I can successfully PXE boot. I just don’t create the RamDiskTFTPBlockSize so it should use the default value of 1024.

    I have found out something really interesting. If I use the build-in SCCM boot images instead of the MDT boot images it goes much faster even though they have the same size…

    Reply
    • Brent
      Nov 12 2014

      Ok, I assume you can no longer successfully PXE boot at all (like me) if you don’t allow RamDiskTFTPBlockSize to be default value of 1024? (ie. create and set value to 2048, 4096, etc.)

      Interesting that SCCM boot images would be different than MDT, I wouldn’t think they would be since PXE is just using TFTP to move raw data. I don’t have MDT setup in my environment, so I can’t compare.

      Reply
      • Nov 12 2014

        I can still PXE with some different values but it gets even slower.

        It’s weird indeed the difference between times…

        Are your DPs setup as standalone site systems or are they site servers?

        Reply
        • Brent
          Nov 13 2014

          Interesting, when I use other values it comes back at the same speed just corrupt and fails. My DPs at one site is a site system, my DP at another site is standalone, same results on both.

          Reply
  3. Patrick
    Nov 25 2014

    What VMware answered is right.
    Most of the PXE clients out there do not support jumbo frames, IP fragmentation and the like.

    TFTP is originally a lock-step protocol that for every data block transferred the DataSender stops for the ACK coming from the DataReceiver.
    If the round trip delay is big, regular TFTP transfers get negatively affected. The TFTP protocol originally ruled a block size of only 512 bytes, next the negotiated “blocksize” option extension allowed to set block sizes of any size but limited to 1468 bytes in Ethernet environments with MTU=1500.
    When you guys “touch” RamDiskTFTPBlockSize is to set a bigger block size in order to need fewer ACKs “but” you also get the unwanted IP fragmentation side effect. Not good.
    To really solve this problem you guys have to deal with the “windowed” TFTP concept; the idea is the DataSender sends “windowsize” consecutive data blocks before to stop for a “single” ACK. This way we are able to avoid IP fragmentation (blocksize <= 1468) and we need to wait for fewer ACKs what implies faster transfers.
    Microsoft implements 2 schemas with fixed and variable "windowsize" and that can be set at bcd level.

    Best,
    Patrick

    Reply
    • Brent
      Dec 1 2014

      Patrick, sorry for the slow reply over the holiday. Thanks for the great information and insight. We actually did try making adjustments to the window sizes without any positive improvements. We have the setting “Enable Variable Window Extension” turned on in Windows TFTP (variable window size) and it doesn’t help our slow clients. We also tried hard coding a fixed window sizes, and also saw no changes.

      The reason we moved to changing RamRiskTFTPBlockSizes is because it was the only setting that made a positive difference on the slow clients. It seems, that while IP fragmentation is not ideal, most PXE clients (HP, Dell, Wyse, ASUS, Microsoft, etc.) seem to handle it fine, just not for VMware.

      Any ideas why changing the Window Sizes didn’t help our slow PXE clients? In VMware the speed seems to be the same no matter what either setting is (but gets corrupt when IP Fragmentation occurs).

      Reply
      • Patrick
        Dec 2 2014

        At this point you should use a sniffer (Wireshark) running at your server and see what’s really going on at packet level.
        I would not rely on RiskTFTPBlockSizes; I’d just try to analyze what happens when using windowsize 4 or 8 . The wireshark traffic capture will surely help.
        Some idea that comes to my mind is that when you rely on big RiskTFTPBlockSizes the number of data blocks get reduced but when you rely on a windowsize strategy the number of data-packets remains the same only the number of ACKs get reduced, that tells me the delay is produced while transferring the data-packets; that could be the sign of some contrack/firewall layer adding an extra delay for every packet transferred. Please take a look at RFC1350 and see that TFTP traffic requires UDP port 69 open but also a random port used by the server where the actual data-transfer takes place (similar to FTP). If the traffic on this port is for some reason delayed by any intermediate security/track control layer you will see exactly the problem that you describe here. For this reason you must be sure i.e. there’s no any NAT service between your TFTP client and Server. I hope you get the idea.

        Best,
        Patrick

        Reply
        • Brent
          Dec 9 2014

          Patrick, sorry for the delay to complete your suggestion. I ran some Wire Shark captures and don’t really see a clear cause of any problems besides that it is just slow. Below is a relevant snippet of the capture.

          Essentially, the server just repeats Data Packet followed by Acknowledgement until it gets to 65535 and then that counter starts over until finally finishes. This test took over 6 minutes to transfer a file under 200 MB. There is one failure at the top, but it immediately retries and works. Any thoughts?

          No. Time Source Destiniation Protocol Length Info
          1893 41.072709000 CLIENT SERVER UDP 389 Source port: 4011 Destination port: 4011
          1896 41.225051000 SERVER CLIENT UDP 472 Source port: 4011 Destination port: 4011
          1917 42.117013000 CLIENT SERVER TFTP 95 Read Request, File: smsboot\x64\wdsmgfw.efi, Transfer type: octet, tsize\000=0\000, blksize\000=1468\000
          1918 42.117542000 SERVER CLIENT TFTP 71 Option Acknowledgement, blksize\000=1468\000, tsize\000=1459552\000
          1919 42.117958000 CLIENT SERVER TFTP 72 Error Code, Code: Option negotiation failed, Message: User aborted the transfer
          1920 42.157829000 CLIENT SERVER TFTP 87 Read Request, File: smsboot\x64\wdsmgfw.efi, Transfer type: octet, blksize\000=1468\000
          1921 42.158280000 SERVER CLIENT TFTP 57 Option Acknowledgement, blksize\000=1468\000
          1922 42.158605000 CLIENT SERVER TFTP 60 Acknowledgement, Block: 0
          1923 42.158970000 SERVER CLIENT TFTP 1514 Data Packet, Block: 1
          1924 42.159401000 CLIENT SERVER TFTP 60 Acknowledgement, Block: 1
          1925 42.159459000 SERVER CLIENT TFTP 1514 Data Packet, Block: 2
          1926 42.159902000 CLIENT SERVER TFTP 60 Acknowledgement, Block: 2
          1927 42.159939000 SERVER CLIENT TFTP 1514 Data Packet, Block: 3
          1928 42.160329000 CLIENT SERVER TFTP 60 Acknowledgement, Block: 3

          297793 507.978765000 SERVER CLIENT TFTP 1478 Data Packet, Block: 65535
          297794 507.981821000 CLIENT SERVER TFTP 60 Acknowledgement, Block: 65535
          297795 507.981922000 SERVER CLIENT TFTP 1478 Data Packet, Block: 0
          297795 507.981922000 SERVER CLIENT TFTP 1478 Data Packet, Block: 0

          349347 613.783380000 SERVER CLIENT TFTP 558 Data Packet, Block: 60
          349348 613.783691000 CLIENT SERVER TFTP 60 Acknowledgement, Block: 60
          349349 613.783750000 SERVER CLIENT TFTP 86 Data Packet, Block: 61 (last)
          349350 613.784081000 CLIENT SERVER TFTP 60 Acknowledgement, Block: 61

          Reply
  4. Patrick
    Dec 10 2014

    what you have done is just a test of a plain RFC1350 transfer without windowsize. To fully understand the capture I’d should see the timing of the packets that is missing in your posting. upload that capture somewhere and I can take a look at it.
    try to run a capture with windosize =4 or 8 just to compare.
    I usually transfer ~200 MB wim files with winsize=8 in about 30/40 secs

    Reply
    • Brent
      Dec 10 2014

      Patrick, I have uploaded the full capture here: (http://1drv.ms/138j59v), let me know once you have looked at it

      Your reply actually makes a lot more sense after some further research. Since we are using System Center Configuration Manager 2012 R2 on top of our WDS server, apparently there are some differences with the TFTP client used (https://social.technet.microsoft.com/Forums/en-US/ae6121d2-18ea-40d4-b662-a2a2ec7d1f18/slow-pxe-boot-tftp-windowing-issue?forum=configmanagerosd). There is supposed to be a DWORD registry key under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP called RamDiskTFTPWindowSize, that I can set to either 4 or 8. Apparently this key does not work. The RamDiskTFTPBlockSize DWORD at this same location does work, but leads to issues as we know.

      One thing about the Configuration Manager implementation is that default.bcd does not exist, so I can’t just force it manually at BCD level. Configuration Manager internally creates a temporary BCD file for each and every PXE request (I can view these temp BCD files and can confirm when RamDiskTFTPBlockSize changes, but RamDiskTFTPWindowSize is always missing).

      Therefore, all this time, since I had this registry key set, I thought I was using a window size of 8, until we ran the capture, which tells us we are not using a WindowSize. Best I can tell from this information, there is no way I can use a WindowSize here with Configuration Manager?

      Also, still not sure why some hardware is slow and some is not when we are not using WindowSizes (RealTek NICs seem to be the slowest and unusable, but other NICs like VMware are acceptable without WindowSizes (about 1-2 minutes)?

      Thanks!!

      Reply
  5. Patrick
    Dec 11 2014

    I studied your capture; taking randomly the round-trip delay it looks consistent around 3.2 mS with a payload of 1432 bytes then considering the transfer (boot.wim) takes 527 seconds from Wireshark we have; 527 s / 0.0032 s * 1432 Bytes = 224 MB sure consistent with the size of a Boot.wim today. Then your TFTP is OK, it’s consistent, but slow… You asked why? why this only happens on some devices? well there are cards that run at 1GB and cards that cannot negotiate that speed, plus there are some with faulty or outdated PXE firmware; that’s it.
    I read the link and it seems SCCM is not adding the windowing parameter to the BCD; the guy that says that the SCCM client doesn’t deal with windowed TFTP is mistaken. It is the bootmgr the one that deals with those TFTP options from the client and the SCCM bootmgr has the windowed TFTP options.
    Your goal now is making SCCM honoring the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP\RamdiskTFTPWindowSize that should trigger the adding of the corresponding parameter to the BCD.

    If you want to know in advance if a windowed TFTP schema will fix your problem take a look at Serva; http://vercot.com/~serva/an/WindowsPXE1.html
    You will be able to quickly test a Windows install with different values of windowsize; the difference is really big.

    Reply
  6. Patrick
    Dec 11 2014

    if you want to go a bit deeper on this try to see which process (name of the dll) is touching RamDiskTFTPBlockSize when creating the BCD; do it with Rusinovich’s procmon or something like that.

    Reply
    • Brent
      Dec 11 2014

      Patrick, thanks, that all makes a lot of sense. I figured the root issue was likely hardware related (bad PXE firmware, etc.); unfortunately this issue is relatively wide spread across cards and hard to just avoid them. It does seem like RamdiskTFTPWindowSize will solve the issue, if can get that feature working with Configuration Manager. Have you ever seen it work with ConfigMgr? I reached out to Microsoft to see if any feedback from them on why the registry key does not work, will see what I hear back from them. I also ran procmon to see what is creating the BCD, looks like its a combination of svchost.exe and wmiprvse.exe; which doesn’t help much…

      Reply
  7. Patrick
    Dec 12 2014

    OK, in SCCM the BCD is created by \BIN\I386\smspxe.dll or \BIN\X64smspxe.dll. I have studied that dll and it is true; while it parses RamDiskTFTPBlockSize it never parses RamdiskTFTPWindowSize. While I can clearly see the code that injects RamDiskTFTPBlockSize in the BCD the code that injects RamdiskTFTPWindowSize is obviously missing.
    I have no a running SCCM handy now then if you have a minute just try this.
    1) back-up your BIN\X64\smspxe.dll
    2) open it with a hex editor like Hex Workshop
    3) search the “unicode” string “((ULONG)(0x000007)” w/o quotes, it should appear only once.
    4) replace the char “7” with an “8” and save the change.
    5) Set the RamDiskTFTPBlockSize registry key to 8
    6) Boot a client.

    What you just did was a dirty patch of the dll that will read RamDiskTFTPBlockSize from the registry but it will inject its value in the BCD as the corresponding RamdiskTFTPWindowSize parameter.
    Now the client will receive in the BCD RamdiskTFTPWindowSize=8 and it’ll take a default for the missing RamDiskTFTPBlockSize value.
    If anything goes wrong just replace the patched dll with the back-up and delete the RamDiskTFTPBlockSize registry key. No big deal

    Let me know if it works.

    Best,
    Patrick

    Reply
  8. Patrick
    Dec 12 2014

    a little change..
    the patch described before only changes the logging of the operation
    for really patching the “action” try these steps instead
    3) search the hexadecimal sequence 07000035, it should appear only once.
    4) replace the byte 07 with 08 and save the change.

    Best,
    Patrick

    Reply
    • Brent
      Dec 12 2014

      Ok, I was just about to reply stating it didn’t work 🙂 I confirmed by looking at file locks, it appears in my environment (x64 OS) that the x64 version of smspxe.dll is used. Should both changes be made, or just your second change? I’ll test it now. Thanks.

      Reply
      • Brent
        Dec 12 2014

        Patrick, also, I just searched for 07000035 and actually got two instances found. Which one needs to be changed? When viewing the hex, should be in Unicode or Legacy ASCII?

        Reply
  9. Patrick
    Dec 12 2014

    I have the version 5.0.7804.1000 and it has only 1 instance.
    (not installed)

    in your case (without seeing the code) I would try changing one, the other one and both instances…

    Reply
    • Brent
      Dec 12 2014

      My version is 5.0.7958.1401. In case you would like to look at it; I put a copy of the DLL here: http://1drv.ms/1zHHLlb

      I will start testing each value

      Reply
  10. Patrick
    Dec 12 2014

    in your case is the first instance; the sequence to find is BA07000035

    Reply
    • Brent
      Dec 12 2014

      This works! (I had just finishing trying both values and can confirm this value is the one that works as well). This makes massive improvements in the TFTP transfer time. My VMware VMs went from 1 minute 30 seconds to 30 seconds. My Hyper-V VMs went from 45 seconds to 15 seconds! My RealTek NICs in a HP box went from 6-8 minutes to 1 minute and 30 seconds (much better and an acceptable time, not as nice as other NICs but definitely useable now). This looks like a major oversight by the ConfigMgr development team to not have this configurable or at the very least set to a variable window size by default (especially since not an issue like this with WDS). At least this dll hack gives a valid workaround until Microsoft fixes it. Thank you very much for your insight! I don’t think I would have been able to find full fix on my own. Btw, how did you find which Hex value in the DLL to update using Hex Workshop? Most of the dll is not readable.

      Reply
  11. Chris
    Dec 12 2014

    Guys, this is a fantastic conversation. I am having almost the exact same problem Brent originally had, with the added oddity that when the VMXNET3 VMs fail, I can almost always get it to work with one or two soft reboots.

    Brent — I have made no changes to anything to correct this. Can you summarize the net changes you made to get it to work? I see the DLL editing, but would we also need to adjust certain registry keys? I assume this is all being done to the SCCM DP?

    This would make a great instructional blog post.

    Thanks very much — I’m hopeful this will correct our problem!

    Reply
    • Brent
      Dec 12 2014

      Chris, many changes were made during testing to narrow stuff done, but to actually reproduce the DLL fix scenario from a default environment is only a couple of steps. Make the following changes on the SCCM DP:

      1) Modify DLL as mentioned in Patrick’s reply (modify HEX sequence of BA07000035 to BA08000035)
      2) Create the following DWORD registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP\RamdiskTFTPBlockSize and set the value to 4 or 8 (8 is faster).

      Note: Setting the labeled BlockSize key in registry is intentional, as this is key the smspxe.dll is looking for, the DLL hack changes how that key is interpreted.
      One other note: When editing the DLL, make sure Windows Deployment Services is stopped, or saving of the DLL will fail with no error.

      That’s it, you should be able to test using just those two changes steps. Let me know if any further questions. I will post back what I hear from Microsoft as I hear.

      Reply
      • Chris
        Dec 12 2014

        🙂 I actually just meant could you explicitly say what the values you are using are, but your post was much better for the misunderstanding lol!

        Thanks very much! Changes made — my problem is somewhat intermittent so it’ll take some testing to know for sure, but fingers crossed!

        It was fun watching this play out today, here and in the TechNet post… especially since I’ve been fighting this for a couple weeks now. You should add the DLL details to your Connect posting too… might get the attention of the product team guys easier. Could just be an oversight in the code.

        Again — great job Brent and Patrick!

        Reply
        • Chris
          Dec 12 2014

          Okay I’m tired and that last post was technically meant for the TechNet thread, since that’s where the misunderstanding was. 🙂

          Setting the WindowSize to 8 made my VMXNET3 VMs do the TFTP download reeeeeaaaalllly slowly, but setting it to 4 made them go really fast and, at least for now, consistently without error!

          Reply
          • Patrick
            Dec 13 2014

            OK A few words about windowsize values.
            when we set windowsize=n the TFTP server will send n consecutive blocks before stopping for an ACK. bootmgr.exe is able to handle the load but remember when bootmgr.exe runs it still relays on the UNDI pxe driver which might have some problems dealing with the burst of TFTP blocks. This situation can be seen specially in pure virtualized environments when the packets never touch a real wire (very fast transfers) and/or the TFTP server runs in fast hardware and it does not control the minimum inter-packet dely. Then a higher windowsize value in these situations could lead to errors and that to slower transfers.
            There’s also the false idea that windowsize can only be 4 or 8, but windowsize can be any value <= 64 (limited by the client).
            Care must be taken when testing high values if the server does not limit the minimum inter-packet delay; this might lead to net resource starvation.
            Fortunately there's always a sweet spot for the windowsize value you just have to test and find it.

            Best,
            Patrick

        • Brent
          Dec 15 2014

          Thanks. I also updated the connect post with further information. Hopefully someone at Microsoft will respond. In addition, I have a support case on the matter as well.

          Reply
  12. Patrick
    Dec 12 2014

    I’m glad it did work. Please consider this is just for testing purposes absolutely not for production. The idea was just to prove that the SCCM client TFTP capabilities “are” really there; it is eventually MS who should add the TFTP windowsize option to smspxe.dll.

    Best,
    Patrick

    Reply
    • Brent
      Dec 12 2014

      Yup, this does tell us that there is no reason for the functionality to not be there. Microsoft just needs to add some very simple code to address it. I have reached out to Microsoft about this and hopefully can get them to fix/add this, just not sure how long that could take…Thanks.

      Reply
  13. Dec 15 2014

    I can confirm it works for me as well. Amazing work, thanks!

    Reply
    • Brent
      Dec 15 2014

      Great, glad to hear that others are seeing the same results as well.

      Reply
  14. P. Cruiser
    Dec 19 2014

    When using Serva as a test, should I be changing the “Enforce Windowed TFTP, window-size” or “Parse ‘windowsize’ option limited to” ?

    Reply
    • Patrick
      Dec 22 2014

      In Serva the windowsize parameter is controlled by
      “Parse ‘windowsize’ limited to”
      possible values e.g. from 4 to 32.

      Reply
  15. Dec 21 2014

    Hi Brent

    Have you tried to make changes to the TFTP tab of the WDS properties?

    There is an option to enable or disable Variable Window Extension.

    Cheers
    Felipe

    Reply
    • Patrick
      Dec 22 2014

      if the code that should parse RamdiskTFTPWindowSize and inject its value in the dynamically created BCD is missing from smspxe.dll then no matter how many settings you might try; the created BCD will not include the windowsize parameter.
      I have “grepped” the whole content of mu_system_center_2012_configuration_manager_and_endpoint_protection_with_sp1_x86_x64_dvd_1565907.iso and there’s not any component containing the string “RamdiskTFTPWindowSize”.

      Best,
      Patrick

      Reply
    • Brent
      Dec 22 2014

      Yes, I can confirm what Patrick already stated, changing that setting in WDS has no affect because BCD is handled by SCCM and smspxe.dll unfortunately does not parse the setting.

      Reply
  16. Brent
    Dec 22 2014

    FYI to all, I got a reply from Microsoft on this issue. They officially have this classified as a “bug” with a DCR assigned and slated to be fixed in SCCM 2015. Unfortunately it looks like there will not be a fix in a SCCM 2012 CU. For now, the DLL hack is the only workaround until they update their code in the next version.

    Reply
    • Chris
      Dec 22 2014

      Still, good job to both of you for finding it!

      Reply
    • Chris
      Feb 9 2015

      Brent – would you mind emailing me your MS case number for that if you have one? I’d like to use it as a reference when I try to get them to issue us a private hotfix. There’s a chance for that given our size and the impact. Thanks!

      Reply
      • Brent
        Feb 10 2015

        Chris, the MS case number was: 114121112156755. Let me know if you make any progress in getting a private fix.

        Reply
        • Patrick
          Feb 10 2015

          I’m also interested on this; if you get the fix please let me know (see my e-address at Serva’s site).

          Best,
          Patrick

          Reply
          • Juha-Matti
            Feb 25 2015

            I am also interested in finding a solution by Microsoft to this.

            I implemented the DLL-changes according to Patrick and it speeded up the boot.wim transfer from 5 minutes to 1 minute 20 seconds. Still it could be faster, but this already is a great enhancement to the situation before.

            In my organisation the optimal WindowSize was 8. WindowSize 4 was 1min 40sec, 16 was 1min 30sec.

            BR,
            Juha-Matti

          • Patrick
            Feb 26 2015

            Fore more details about windowsize behavior and its values you can take a look at RFC 7440
            https://tools.ietf.org/rfc/rfc7440.txt

            Best,
            Patrick

  17. Mar 19 2015

    I’ve just made this change on 2012 R2 CU4.. setting the value to 8 did get poor performance, but 4 is quick 🙂

    Reply
  18. Brent
    Jun 2 2015

    FYI, I just tested this with ConfigMgr 2012 R2 SP1 (2012 SP2) and it appears that Microsoft has not yet fixed the issue by updating the DLL to be able to natively set the WindowSize via the registry. Also, it does appear that Microsoft has modified the DLL in 2012 R2 SP1 (2012 SP2), as the DLL I modified for 2012 R2 no longer works. I had to edit the dll manually again and create a new custom DLL based off the new original to make this fix work in 2012 R2 SP1 (2012 SP2). Searching for the same text in the hex editor as I did for the R2 dll did work.

    Reply
  19. Jul 28 2015

    Hi

    Thanks I have come across your post and have not made the above changes as of yet. The issue I am having is only on one site that has a local DP however when PXE booting I am getting the Contacting Server : IP_Address . . . . . . . . .
    This is intermittent after a few restarts it will eventually give you F12 option.
    Looking in the PXE log all i get is below when I get this issue.

    In SSL, but with no client cert SMSPXE 28/07/2015 22:53:47 4316 (0x10DC)
    Client lookup reply:
    SMSPXE 28/07/2015 22:53:50 4316 (0x10DC)
    00:50:56:A9:AD:27, EBA32942-8C41-5075-E308-D7F8B247847B: device is not in the database. SMSPXE 28/07/2015 22:53:50 4316 (0x10DC)
    Looking for bootImage CE2002D5 SMSPXE 28/07/2015 22:53:50 4316 (0x10DC)
    In SSL, but with no client cert SMSPXE 28/07/2015 22:53:50 4316 (0x10DC)
    In SSL, but with no client cert SMSPXE 28/07/2015 22:53:52 4316 (0x10DC)
    Looking for bootImage CE2002D5 SMSPXE 28/07/2015 22:54:05 4316 (0x10DC)
    In SSL, but with no client cert SMSPXE 28/07/2015 22:54:05 4316 (0x10DC)
    In SSL, but with no client cert SMSPXE 28/07/2015 22:54:07 4316 (0x10DC)
    Looking for bootImage CE2002D5 SMSPXE 28/07/2015 22:54:24 4316 (0x10DC)
    In SSL, but with no client cert SMSPXE 28/07/2015 22:54:24 4316 (0x10DC)
    In SSL, but with no client cert SMSPXE 28/07/2015 22:54:27 4316 (0x10DC)

    Your help will be greatly appreciated

    Reply
  20. Brent
    Aug 5 2015

    FYI, if anyone is using the custom DLL, I found one caveat with upgrading to SCCM 2012 R2 SP1 CU1. If you have the custom DLL in place on a remote distribution point, the Site Server will fail to update the DP on any remote server that is using the custom DLL, noting that it can’t validate the smspxe DLL. Before install CU1 it is important to remove the custom DLL, replace with the original one, upgrade to CU1, then put the custom DLL back in place.

    Reply
  21. Erik
    Sep 7 2015

    sorry for bumping a post this old, but i cant find any answers.

    We are running configmgr 2007 R3. After migrating confmgr to co-location and wmvare hosts the tftp transfer takes forever.

    I´m trying to set RamDiskTFTPBlockSize to 1432 (thats what supported in vmware) in the registry and then I am changing smspxe.dll (X:\sms_ccm\x64\smspxe.dll) from BA07000035 to BA08000035 but it doesnt go any faster.

    And if I check one of the autogenerated bcd files in smstemp it says this:

    Setup Ramdisk Options
    ———————
    identifier {ramdiskoptions}
    ramdisksdidevice boot
    ramdisksdipath \SMSBoot\boot.sdi
    ramdisktftpwindowsize 1432

    Any help whit this?

    Reply
    • Patrick
      Sep 9 2015

      1) you should set RamDiskTFTPBlockSize to 8 (or 4) in the registry.
      2) The created BCD should then include ramdisktftpwindowsize = 8 (or 4)
      3) Now you boot a client.

      Clients will limit ramdisktftpwindowsize to 64 if the BDC offers a bigger number. But a big number like 64 can many times lead to slow transfers. Please read the thread again.

      Best,
      Patrick

      Reply
      • Erik
        Sep 10 2015

        Yeah, i read it again after i posted and changed to 8, the result is slightly better, but it still takes around 15min. Think we might have package drops, will investigate further later when i have time

        Reply
        • Patrick
          Sep 10 2015

          1) TFTP needs a good network.
          2) you can also trim the value; probably 4 will lead to better results than 8.

          Reply
  22. Sep 9 2015

    I’m using vmware here and I have set RamDiskTFTPBlockSize to 8

    To confirm, you are using the VMXNet3 card in the vm settings?

    Reply
    • Sep 9 2015

      Just to also add that, its set to that as the after the hex edit the value is actually used as the TCPWindowSize and as the param isnt specified for the block size it defaults to an internal value around 1432 iirc.

      Reply
    • Patrick
      Sep 9 2015

      This mod should not be affected by the used NIC driver at all.
      It is the client’s bootmgr.exe/bootmgfw.efi the one that will interpret the BCD parameter ramdisktftpwindowsize and act accordingly when TFTP transferring i.e. Boot.wim

      You are correct about adding RamDiskTFTPBlockSize to the registry in order to get its value added to the BCD as “TCPWindowSize” while the missing RamDiskTFTPBlockSize is replaced at the client by a default value.

      Best,
      Patrick

      Reply
      • Sep 14 2015

        The only reason I state VMXNet3 is that I only reliably had the issue after changing my vm’s from the E1000 to the VMXNet3 driver – so something was worse when using the VMXNet3 🙂

        Reply
        • Patrick
          Sep 14 2015

          I see, well VMware has some issues with TFTP.
          i.e. you can see a report here
          https://communities.vmware.com/thread/519250?sr=inbox
          about how their new Worstation 12 client mistakenly set the TFTP block size “only” when using particular drivers. This thing started happening with Workstation 12.

          You might be running into some related issue;

          Reply
  23. sebus
    Oct 7 2015

    Following Bret info:

    1) Modify DLL as mentioned in Patrick’s reply (modify HEX sequence of BA07000035 to BA08000035)
    2) Create the following DWORD registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP\RamdiskTFTPBlockSize and set the value to 4 or 8 (8 is faster).

    makes my Vmware VM not boot (it just hangs on .sdi load)

    The

    Reply
  24. sebus
    Oct 7 2015

    I can see the bcd used is (due to edit of the default.bcd):

    Device options
    ————–
    identifier {68d9e51c-a129-4ee1-9725-2ab00a957daf}
    ramdisksdidevice boot
    ramdisksdipath \Boot\Boot.SDI
    ramdisktftpblocksize 1456
    ramdisktftpwindowsize 8
    ramdisktftpvarwindow Yes

    Reply
  25. sebus
    Oct 7 2015

    And the above works, but slow

    Reply
  26. i3laze
    Nov 6 2015

    Awesome thread!

    Just patched smspxe.dll from BA07000035 to BA08000035 and set RamDiskTFTPBlockSize=1432 globally.

    Did some tests (BlockSize is set to 1432 in both cases)
    Original dll:
    SCCM boot.wim (250 Mb) is downloaded over 100Mpbs in 55 sec.
    Modified dll:
    Same file on the black screen is downloaded in 25 sec.

    Replaced it everywhere via package (.cmd):

    net stop WDSServer
    IF EXIST C:\SMS_DP$ copy smspxe.dll C:\SMS_DP$\sms\bin /y
    IF EXIST D:\SMS_DP$ copy smspxe.dll D:\SMS_DP$\sms\bin /y
    IF EXIST E:\SMS_DP$ copy smspxe.dll E:\SMS_DP$\sms\bin /y
    IF EXIST “C:\Program Files\Microsoft Configuration Manager\bin\X64” copy smspxe.dll “C:\Program Files\Microsoft Configuration Manager\bin\X64” /y

    reg add HKLM\SOFTWARE\Microsoft\SMS\DP /v RamDiskTFTPBlockSize /t REG_DWORD /d 1432 /f

    net start WDSServer

    Reply
    • Patrick
      Nov 7 2015

      @i3laze
      you are making a mistake:
      After the patch the RamdiskTFTPBlockSize variable will in fact be interpreted by mspxe.dll as “RamdiskTFTPWindowSize” then RamdiskTFTPBlockSize should be set to any value between 4 and 8 or between 4 and 16.

      If after the patch you do set RamdiskTFTPBlockSize = 1432 the booting client will limit RamdiskTFTPWindowSize to 64 but that high value many times leads to suboptimal transfer figures.

      Best,
      Patrick

      Reply
  27. i3laze
    Nov 9 2015

    I was misleaded by tons of “RamDiskTFTPBlockSize=1432” in the comments above.

    So you suggest to
    A. Remove the key from registry?
    or
    B. Patch it to 8?

    Reply
    • Patrick
      Nov 9 2015

      just set RamDiskTFTPBlockSize=8 or 4 or 16 and see the results.

      When you set RamDiskTFTPBlockSize=1432 the client was limiting the value to 64, if you saw transfer improvements in this situation then your network is good and windowsize=64 is not bad for you but this is just “your” case. For a more conservative approach just use 4/8/ or 16

      Best,
      Patrick

      Reply
  28. i3laze
    Nov 10 2015

    Got ~25sec at 8 and ~20sec at 16.

    I’ll stay at 16, cause performance is fine.

    reg add HKLM\SOFTWARE\Microsoft\SMS\DP /v RamDiskTFTPBlockSize /t REG_DWORD /d 16 /f

    Reply
  29. sebus
    Nov 14 2015

    RamdiskTFTPBlockSize = 16 with hacked dll and corresponding setting in bcd, makes 450 Mb .wim boot in Optiplex 3020 in 10 seconds.

    Same wim booting Hyper-V 2012 R2 VM with legacy netowork (only one that can do PXE boot) takes > 1 HOUR !!!

    sebus

    Reply
  30. i3laze
    Nov 19 2015

    Was updating from 2012R2 SP1CU1 to CU2.

    Got upgrade error on all DPs : )

    distmgr.log says:
    “Error in verifying the trust of file ‘\\DistrPoint.FQDN.COM\SMS_DP$\sms\bin\smspxe.dll’.
    Failed to install DP files on the remote DP. Error code = 16389”

    Have to roll back original CU1 smspxe.dll for a while.
    Just FYI.

    Reply
    • Brent
      Nov 19 2015

      Ahh yeah, I ran into that issue too, I meant to post an update here but I forgot. You have to roll DLL back before doing any SCCM upgrades, otherwise it will fail. Once the upgrades are complete you can replace it again. If smspxe.dll got changed by the update, then you need to edit it again, otherwise you can use the same edited DLL.

      Reply
  31. i3laze
    Nov 20 2015

    Got a collection with SCCM Distribution Points for this case : )

    select * from SMS_R_System where SMS_R_System.ResourceNames in (Select ServerName FROM SMS_DistributionPointInfo)

    I’ve created app deployment for my script above.
    Detection method is “modify date le 11/01/2015 for smspxe.dll in 4 places C:\SMS_DP$ or D or E or C:\ProgramFiles paths”.
    And another app deployment with script reverting original file back.

    Reply
  32. Blade
    Dec 2 2015

    Not sure what is going on but after I upgraded to SCCM 2012 R2 SP1 and modified the DLL on my remote site it hangs on the SDI file when I am trying to image my Dell boxes. I have to put the original back which is taking about 15 minutes to complete. Has any one else had this problem

    Reply
    • Blade
      Dec 2 2015

      Sorry both my Block and Windows are set to 8

      Reply
      • Patrick
        Dec 2 2015

        try with different clients; On rare occasions some clients present problems with windowed TFTP. Also check your traffic with wireshark and try to see how the sdi TFTP transfer fails.

        Best,
        Patrick

        Reply
  33. Brent
    Mar 18 2016

    FYI, Microsoft has finally fixed this. Its available in Technical Preview 1603 of Configuration Manager and should also be in the next Current Branch release as well (after 1602). More info: https://technet.microsoft.com/en-us/library/mt595861.aspx#BKMK_RC1603

    Reply
    • Patrick
      Mar 18 2016

      Is it confirmed? It looks like they finally did it; good job MS.

      Reply
      • Brent
        Jul 25 2016

        I can confirm this is out of preview and in version 1606! I have tested it and works great! Yay! 🙂

        Reply
  34. Mitch
    Jul 25 2016

    How do we use this in 1606? Is there a place in the console to enable it or tweak the settings?

    Thanks.

    Reply
    • Brent
      Jul 25 2016

      Unfortunately there are no settings viewable in the UI. This is fully configurable via the registry. Simply open the Registry Editor, navigate to the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP. Once there create a new DWORD for which setting you want to manage. For Window Size: RamDiskTFTPWindowSize or Block Size: RamDiskTFTPBlockSize. Then enter the size you want for the Window/Block sizes for the value field. I have found leaving RamDiskTFTPBlockSize default and changing RamDiskTFTPWindowSize to either 6 or 8 works best in my environment.

      Reply

Trackbacks & Pingbacks

  1. PXE Boot Optimierung nach SCCM 1606 Update – SCCMing
  2. PXE Boot Optimierung – regsvr32 /i kaos.dll

Share your thoughts, post a comment.

(required)
(required)

Note: HTML is allowed. Your email address will never be published.

Subscribe to comments