I got some serious troubles setting up RRAS.

First of all - i can't ping my server.
My Mailserver(hMailServer) can't sent out mails using SMTP, even though TCP PORT 25 is opened.
FTP isn't working properly - i can log in, but i can't see any folders or files.
NSLookUp is returning a timeout. (port 53 TCP/UDP is open)

All of these troubles disapear when i turn of RRAS - what am i doing wrong?

Would it be easier setting up IPSec?

I'm not exactly sure how you've configured RRAS, but here are some step-by-step instructions I wrote awhile back.

1. Click Start
2. Click Administrative Tools
3. Click Routing and Remote Access
4. In the left-hand menu, select NAT/Basic Firewall under (Machine name)->IP Routing
5. Right-click Local Area Connection (whichever is the WAN connection)
6. Click Properties
7. Click Inbound Filters
8. Click New...
9. Check the Source network checkbox
10. Enter "12.96.160.0" for the IP address
11. Enter "255.255.255.0" for the Subnet mask
12. Click OK
12. Repeat steps 8-12 with the other address (67.19.0.0/255.255.255.0 and 70.84.160.0/255.255.255.0)
13. Click OK
14. Click Apply
15. Click OK
16. Close Routing and Remote Access

You will want to select the "Drop all packets except those that meet the criteria below" option.

You will also need to add all the ports or whitelisted IP networks you want open on the server.

well...

after following your guide AND applying the same rules for outbound(ofcourse just as destination) works, the DNS-lookup works, but all of the other problems still exists...

Can you post a screen shot of your configuration?

It's a little bit difficult to get a complete view - but with a couple of screen-shots i hope you get the idea.
The services/ports-config:


The Outbound IP-Packet-filter:



The inbound is just opposite

Where did you find those options !?!...

No ideas to what i am doing wrong here?

Typically we do not recommend outbound filtering. However, I may be able to provide some additional assistance.

Are you permitting ICMP through the firewall? ICMP is IP protocol 1.

Can you establish outbound connections to tcp port 25 via the command line? Perhaps hMailServer is trying to use 465/tcp for SSL-enabled SMTP.

Are you using Active or Passive FTP? Passive FTP is most common, but requires you to open a range of ports for data transmission. Active FTP typically requires ports 20/tcp (data) and 21/tcp (control).

Hmmm, I haven't opened port 20 and FTP seems to work ok for me

In my experience it isn't required by Windows. Here is some documentation regarding the differences between Passive and Active FTP:

http://slacksite.com/other/ftp.html

Please check it out my post regarding to Full Windows Free Server Firewall with Brute Force Detection in the Hows To and Windows Security Boards

Thanks

Claudio