I would suggest you not remove the remainder of the vinyl.

My tendency toward a solution would be to prime the door. Apply an exterior grade filler, prime the filler and paint. While this will be slightly more involved than written, it is what I would do to affect a repair.

First, ask your local paint store guy, for a bonding type primer. Stay away from Kilz as a fix-all. Prime the entire door so your brush strokes will not be visible in the finish product.

Second, smooth on an exterior grade spackling. Personally I do not find the "Shrink-Free" types easy to use, but be sure it is an exterior use product.

Third, sand the area, check if a second application is necessary for a uniform look.

Last, if need be sand again, prime again and then apply two coats of finish.