JIT compilation and DEP

One possibility is to make it a requirement that Windows installations running your program be either configured for DEP AlwaysOff (bad idea) or DEP OptOut (better idea).

This can be configured (under WinXp SP2+ and Win2k3 SP1+ at least) by changing the boot.ini file to have the setting:

/noexecute=OptOut

and then configuring your individual program to opt out by choosing (under XP):

Start button
    Control Panel
        System
            Advanced tab
                Performance Settings button
                    Data Execution Prevention tab

This should allow you to execute code from within your program that's created on the fly in malloc() blocks.

Keep in mind that this makes your program more susceptible to attacks that DEP was meant to prevent.

It looks like this is also possible in Windows 2008 with the command:

bcdedit.exe /set {current} nx OpOut

DEP is just turning off Execution permission from every non-code page of memory. The code of application is loaded to memory which has execution permission; and there are lot of JITs which works in Windows/Linux/MacOSX, even when DEP is active. This is because there is a way to dynamically allocate memory with needed permissions set.

Usually, plain malloc should not be used, because permissions are per-page. Aligning of malloced memory to pages is still possible at price of some overhead. If you will not use malloc, some custom memory management (only for executable code). Custom management is a common way of doing JIT.

There is a solution from Chromium project, which uses JIT for javascript V8 VM and which is cross-platform. To be cross-platform, the needed function is implemented in several files and they are selected at compile time.

Linux: (chromium src/v8/src/platform-linux.cc) flag is PROT_EXEC of mmap().

void* OS::Allocate(const size_t requested,
                   size_t* allocated,
                   bool is_executable) {
  const size_t msize = RoundUp(requested, AllocateAlignment());
  int prot = PROT_READ | PROT_WRITE | (is_executable ? PROT_EXEC : 0);
  void* addr = OS::GetRandomMmapAddr();
  void* mbase = mmap(addr, msize, prot, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
  if (mbase == MAP_FAILED) {
    /** handle error */
    return NULL;
  }
  *allocated = msize;
  UpdateAllocatedSpaceLimits(mbase, msize);
  return mbase;
}

Win32 (src/v8/src/platform-win32.cc): flag is PAGE_EXECUTE_READWRITE of VirtualAlloc

void* OS::Allocate(const size_t requested,
                   size_t* allocated,
                   bool is_executable) {
  // The address range used to randomize RWX allocations in OS::Allocate
  // Try not to map pages into the default range that windows loads DLLs
  // Use a multiple of 64k to prevent committing unused memory.
  // Note: This does not guarantee RWX regions will be within the
  // range kAllocationRandomAddressMin to kAllocationRandomAddressMax
#ifdef V8_HOST_ARCH_64_BIT
  static const intptr_t kAllocationRandomAddressMin = 0x0000000080000000;
  static const intptr_t kAllocationRandomAddressMax = 0x000003FFFFFF0000;
#else
  static const intptr_t kAllocationRandomAddressMin = 0x04000000;
  static const intptr_t kAllocationRandomAddressMax = 0x3FFF0000;
#endif

  // VirtualAlloc rounds allocated size to page size automatically.
  size_t msize = RoundUp(requested, static_cast<int>(GetPageSize()));
  intptr_t address = 0;

  // Windows XP SP2 allows Data Excution Prevention (DEP).
  int prot = is_executable ? PAGE_EXECUTE_READWRITE : PAGE_READWRITE;

  // For exectutable pages try and randomize the allocation address
  if (prot == PAGE_EXECUTE_READWRITE &&
      msize >= static_cast<size_t>(Page::kPageSize)) {
    address = (V8::RandomPrivate(Isolate::Current()) << kPageSizeBits)
      | kAllocationRandomAddressMin;
    address &= kAllocationRandomAddressMax;
  }

  LPVOID mbase = VirtualAlloc(reinterpret_cast<void *>(address),
                              msize,
                              MEM_COMMIT | MEM_RESERVE,
                              prot);
  if (mbase == NULL && address != 0)
    mbase = VirtualAlloc(NULL, msize, MEM_COMMIT | MEM_RESERVE, prot);

  if (mbase == NULL) {
    LOG(ISOLATE, StringEvent("OS::Allocate", "VirtualAlloc failed"));
    return NULL;
  }

  ASSERT(IsAligned(reinterpret_cast<size_t>(mbase), OS::AllocateAlignment()));

  *allocated = msize;
  UpdateAllocatedSpaceLimits(mbase, static_cast<int>(msize));
  return mbase;
}

MacOS (src/v8/src/platform-macos.cc): flag is PROT_EXEC of mmap, just like Linux or other posix.

void* OS::Allocate(const size_t requested,
                   size_t* allocated,
                   bool is_executable) {
  const size_t msize = RoundUp(requested, getpagesize());
  int prot = PROT_READ | PROT_WRITE | (is_executable ? PROT_EXEC : 0);
  void* mbase = mmap(OS::GetRandomMmapAddr(),
                     msize,
                     prot,
                     MAP_PRIVATE | MAP_ANON,
                     kMmapFd,
                     kMmapFdOffset);
  if (mbase == MAP_FAILED) {
    LOG(Isolate::Current(), StringEvent("OS::Allocate", "mmap failed"));
    return NULL;
  }
  *allocated = msize;
  UpdateAllocatedSpaceLimits(mbase, msize);
  return mbase;
}

And I also want note, that bcdedit.exe-like way should be used only for very old programs, which creates new executable code in memory, but not sets an Exec property on this page. For newer programs, like firefox or Chrome/Chromium, or any modern JIT, DEP should be active, and JIT will manage memory permissions in fine-grained manner.